128 replies to this topic

[doglover]

Let me explain my problem.

 

I grab and upload the EPG data to the distribution servers.  I have 2 PC's for doing this.  On at my house, and one at Rytec's.

The one at rytec's is connected to the internet with a crappy internet line. Upload 0.5 Mbit/s

Historically this PC has been used to distribute all the files.  Which really took a long time.  However my own internet line was not much better at an Upload of 1 Mbit/s

So we kept suffering.

 

Now very recently my home line has been upgraded. (Proximus performed some upgrades on the infrastructure)  Now my line says Upload of 2 mbit/s.  However I see speeds in excess of 3 mbit/s in practice.  This varies however from test to test.  The line does not seem to be capped in anyway.

 

So I am trying now to distribute everything from my home.  But I do not want my PC's to run all day.

One of my linux boxes is running all the time.  And I am investigating the possibility to do the distribution with this linux machine.

 

Therefore I need a command line FTP program for the linux OpenPli box, which I can automate.

So far the answers where not very succesfull.

 

Willy

 

PS: On my windows machines I am using Filezilla, WinSCP and CoreFTP. 

Filezilla does not have an commandline interface.  CoreFTP is a crappy program, and WinSCP works good on most servers, but there is one server which it does not like.  The full program works OK, but in the command line version, I get continously disconnects during an upload to this one server.

Edited by doglover, 16 April 2015 - 10:27.

[betacentauri]

For one box or for all?
For one box download latest busy box binary and put it on the box as extra binary or replace the original. Well better at first put it to /tmp, make it executable and test it.
http://www.busybox.n...inaries/latest/

 

Why not use this?? It works:

root@et7x00:~# wget http://www.busybox.net/downloads/binaries/latest/busybox-mipsel
Connecting to www.busybox.net (140.211.167.224:80)
busybox-mipsel       100% |********************************************************|  1539k  0:00:00 ETA
root@et7x00:~# chmod 755 busybox-mipsel 
root@et7x00:~# ./busybox-mipsel 
BusyBox v1.21.1 (2013-07-08 11:09:23 CDT) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2012.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list[-full]
   or: busybox --install [-s] [DIR]
   or: function [arguments]...

        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as.

Currently defined functions:
        [, [[, acpid, add-shell, addgroup, adduser, adjtimex, arp, arping, ash, awk, base64, basename,
        beep, blkid, blockdev, bootchartd, brctl, bunzip2, bzcat, bzip2, cal, cat, catv, chat, chattr,
        chgrp, chmod, chown, chpasswd, chpst, chroot, chrt, chvt, cksum, clear, cmp, comm, conspy, cp,
        cpio, crond, crontab, cryptpw, cttyhack, cut, date, dc, dd, deallocvt, delgroup, deluser,
        depmod, devmem, df, dhcprelay, diff, dirname, dmesg, dnsd, dnsdomainname, dos2unix, du,
        dumpkmap, dumpleases, echo, ed, egrep, eject, env, envdir, envuidgid, ether-wake, expand, expr,
        fakeidentd, false, fbset, fbsplash, fdflush, fdformat, fdisk, fgconsole, fgrep, find, findfs,
        flock, fold, free, freeramdisk, fsck, fsck.minix, fsync, ftpd, ftpget, ftpput, fuser, getopt,
        getty, grep, groups, gunzip, gzip, halt, hd, hdparm, head, hexdump, hostid, hostname, httpd,
        hush, hwclock, id, ifconfig, ifdown, ifenslave, ifplugd, ifup, inetd, init, insmod, install,
        ionice, iostat, ip, ipaddr, ipcalc, ipcrm, ipcs, iplink, iproute, iprule, iptunnel, kbd_mode,
        kill, killall, killall5, klogd, last, less, linux32, linux64, linuxrc, ln, loadfont, loadkmap,
        logger, login, logname, logread, losetup, lpd, lpq, lpr, ls, lsattr, lsmod, lsof, lspci, lsusb,
        lzcat, lzma, lzop, lzopcat, makedevs, makemime, man, md5sum, mdev, mesg, microcom, mkdir,
        mkdosfs, mke2fs, mkfifo, mkfs.ext2, mkfs.minix, mkfs.vfat, mknod, mkpasswd, mkswap, mktemp,
        modinfo, modprobe, more, mount, mountpoint, mpstat, mt, mv, nameif, nanddump, nandwrite,
        nbd-client, nc, netstat, nice, nmeter, nohup, nslookup, ntpd, od, openvt, passwd, patch, pgrep,
        pidof, ping, ping6, pipe_progress, pivot_root, pkill, pmap, popmaildir, poweroff, powertop,
        printenv, printf, ps, pscan, pstree, pwd, pwdx, raidautorun, rdate, rdev, readahead, readlink,
        readprofile, realpath, reboot, reformime, remove-shell, renice, reset, resize, rev, rm, rmdir,
        rmmod, route, rpm, rpm2cpio, rtcwake, run-parts, runlevel, runsv, runsvdir, rx, script,
        scriptreplay, sed, sendmail, seq, setarch, setconsole, setfont, setkeycodes, setlogcons,
        setserial, setsid, setuidgid, sh, sha1sum, sha256sum, sha3sum, sha512sum, showkey, slattach,
        sleep, smemcap, softlimit, sort, split, start-stop-daemon, stat, strings, stty, su, sulogin,
        sum, sv, svlogd, swapoff, swapon, switch_root, sync, sysctl, syslogd, tac, tail, tar, tcpsvd,
        tee, telnet, telnetd, test, tftp, tftpd, time, timeout, top, touch, tr, traceroute, traceroute6,
        true, tty, ttysize, tunctl, udhcpc, udhcpd, udpsvd, umount, uname, unexpand, uniq, unix2dos,
        unlzma, unlzop, unxz, unzip, uptime, users, usleep, uudecode, uuencode, vconfig, vi, vlock,
        volname, wall, watch, watchdog, wc, wget, which, who, whoami, whois, xargs, xz, xzcat, yes,
        zcat, zcip

root@et7x00:~# ./busybox-mipsel ftpput
BusyBox v1.21.1 (2013-07-08 11:09:23 CDT) multi-call binary.

Usage: ftpput [OPTIONS] HOST [REMOTE_FILE] LOCAL_FILE

Upload a file to a FTP server

        -v,--verbose            Verbose
        -u,--username USER      Username
        -p,--password PASS      Password
        -P,--port NUM           Port

root@et7x00:~# 

[LraiZer]

Yafc was the only command line ftp that worked to solve continous disconnects during uploads to a server in an automated script on my ubuntu build machine.

 

[doglover]

I will try the busybox.  (already downloaded it)  Did not know a mipsel variant existed.  Thought it had to come from the image buidelrs.

 

Unfortunally Yafc is not compiled for mipsel.  And I do not know enough about compiling it myself.

If some-one can do this.  I will try it.

 

Willy

[doglover]

The ftpput (and get) does not do wildcards. So I had to resort to writing a .sh file for doing the job.

However on hindsight this is not a bad thing, as I have much more control on what is going on.

 

To give you an idea about the .sh file:

#!/bin/sh
date
k=1
FILE=/hdd/epg/Filelist.txt
while read line;do
        echo "Line # $k: $line"
        if k < 17 then
            /hdd/epg/busybox-mipsel ftpget -v --user=xxxxxxx --pass=xxxxxxx IP-adres /hdd/epg/$line /ready/$line
        fi
        /hdd/epg/busybox-mipsel ftpput -v --user=xxxxxxx --pass=xxxxxxx1 IP-adres1 /$line /hdd/epg/$line
        /hdd/epg/busybox-mipsel ftpput -v --user=xxxxxxx --pass=xxxxxxx2 IP-adres2 /$line /hdd/epg/$line
        /hdd/epg/busybox-mipsel ftpput -v --user=xxxxxxx --pass=xxxxxxx3 IP-adres3 /$line  /hdd/epg/$line       
        let k=k+1
done < $FILE
date

Executing this, and piping the output to a file, gives me also a nice log file.

 

Willy

 

PS:  the if loop is in there because I do not need to download all the files (only part)

Edited by doglover, 17 April 2015 - 13:21.

[doglover]

Left to done:  one of the servers I can only log into by SFTP.  So find an SFTP application for a mipsel architecture.

Or find somebody which can compile Yafc for me.  (inclusive of the shared libraries)

 

Willy

Edited by doglover, 17 April 2015 - 13:53.

[theparasol]

I just learned about curl while doing some searching for sftp connections.

What even is better: Its standard in openpli images!

Perhaps its a better idea to start using curl as its supports sftp too

 

You can use it like this:

 

curl -T uploadfile -u user:passwd ftp://ftp.upload.com/myfile

 

Read more here: http://curl.haxx.se/docs/manual.html

[athoik]

Left to done: one of the servers I can only log into by SFTP. So find an SFTP application for a mipsel architecture.
Or find somebody which can compile Yafc for me. (inclusive of the shared libraries)

Willy

Did you try scp?

[doglover]

It is not in OpenPLi, but it is in the feed.

So you have to install it.

 

But: curl: (1) Protocol "sftp" not supported or disabled in libcurl

 

Willy                                                   

                                                     
                                                      

[doglover]

 

Left to done: one of the servers I can only log into by SFTP. So find an SFTP application for a mipsel architecture.
Or find somebody which can compile Yafc for me. (inclusive of the shared libraries)

Willy

Did you try scp?

 

 

yes: problems with using logins user/password and login key

 

Willy

[theparasol]

But: curl: (1) Protocol "sftp" not supported or disabled in libcurl                                                   

                                                      

 

Yes, indeed in the feed.... what a bummer!

 

But perhaps a dev is willing to add support for it:

 

http://iamnearlyther...lities-to-curl/

 

It can work, only packages in feed have to be compiled differently.

[doglover]

SCP:  Can you give me an example to upload a file on a sftp server with username, password en rsa-key

And not the printout of the help file:

 

usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]     
           [-l limit] [-P port] [-S program]                                
           [[user@]host1:]file1 [...] [[user@]host2:]file2     

 

I can get this myself.  I only cannot seems to figure out how to construct the command line.

 

Willy          

[doglover]

 

But: curl: (1) Protocol "sftp" not supported or disabled in libcurl                                                   

                                                      

 

Yes, indeed in the feed.... what a bummer!

 

But perhaps a dev is willing to add support for it:

 

http://iamnearlyther...lities-to-curl/

 

It can work, only packages in feed have to be compiled differently.

 

 

This should be helpfull.

 

Willy

[betacentauri]

Afaik you cannot use scp for a sftp server. You can only use scp when you can login via ssh.

 

But you can install sftp command:

root@et7x00:~# opkg install openssh-sftp
Installing openssh-sftp (6.7p1-r0.0) on root.
root@et7x00:~# sftp
usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]
          [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
          [-o ssh_option] [-P port] [-R num_requests] [-S program]
          [-s subsystem | sftp_server] host
       sftp [user@]host[:file ...]
       sftp [user@]host[:dir[/]]
       sftp -b batchfile [user@]host

Edited by betacentauri, 17 April 2015 - 18:09.

[Erik Slagter]

Both scp and sftp use the same underlying transport (ssh). Maybe you are confused with ftps (like many people)?

[athoik]

SCP:  Can you give me an example to upload a file on a sftp server with username, password en rsa-key
And not the printout of the help file:
 
usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]     
           [-l limit] [-P port] [-S program]                                
           [[user@]host1:]file1 [...] [[user@]host2:]file2     
 
I can get this myself.  I only cannot seems to figure out how to construct the command line.
 
Willy

Please have a look here: https://www.linux-ti...rsa-key-problem

Edited by athoik, 17 April 2015 - 20:34.

[catastrofus]

You can use this example for your SFTP command: http://www.unix.com/...9eb01006ecb5dfd

You are a bit confusing in what you want. Giving a scp usage with username/password AND rsa key (?) while you just want to have an example of a sftp (batch) command...

 

SCP:  Can you give me an example to upload a file on a sftp server with username, password en rsa-key

And not the printout of the help file:

 

usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]     
           [-l limit] [-P port] [-S program]                                
           [[user@]host1:]file1 [...] [[user@]host2:]file2     

 

I can get this myself.  I only cannot seems to figure out how to construct the command line.

 

Willy          

Edited by catastrofus, 17 April 2015 - 21:13.

[doglover]

You are a bit confusing in what you want. Giving a scp usage with username/password AND rsa key (?) while you just want to have an example of a sftp (batch) command...

 

 

Yes this is what I want.

But there is here somebody which is trying to let it me do with scp.

 

Willy

[Erik Slagter]

But scp and sftp do the same, they both work on top of ssh. The difference is that scp works in "batch" mode and sftp is interactive, so if you CAN use ssh, I'd recommend to use scp, because you can use it from a script. You can also use wildcards (if properly escaped with quotes) and copy recursively.

[doglover]

OK.  I can log-in with SFTP manually.

Can even make a batch file to transfer files.

 

However it still requires me to type in the password manually.   I cannot find out how to pass the password in the batch file or from the command line.

 

BTW:  this is also the case for scp !!!

sftp -b /hdd/epg/king.txt user@IP

and in king.txt

cd /epg_data/
put /hdd/epg/rytecxmltvxxxxxx.gz

But this requires me to type in the password.  How can I pass this on automatically!!!!!

 

Willy

 

PS: it is obvious now I need an example, and not a referal to the webpage somewhere with some explanation.

I will try to find some examples with google.