A very common scenario when we execute some code on a remote machine using a ssh session is the when the connection terminates the program or script is also killed. A very simple solution is using a utility program called screen. On Ubuntu you can install screen using
$ sudo apt-get install screen
Now all you need to do is to start a screen session on the remote machine. Just type in
in terminal. This will start a new screen session.
Now like what you do in a normal terminal session just run your script say your tensorflow code
$ python seq2seq.py
Every screen session is assigned an ID which we will use later. Now say while your model is learning you want to exit the remote connection. One important thing to do just before exiting is detaching from the screen session
Simply type in ‘ctrl+a‘ followed by ‘d‘, ‘ctrl+a‘ is for entering screens command line mode and ‘d‘ is simply to detach from the current session. Now you can safely exit.
Next time you login typein
$ screen -ls
this will list active screen sessions along with their IDs. To enter a particular session simply typein
$ screen -r ID
$ screen -r 20873.tty2.amitn
which takes you to the selected screen session and voila you have your code running.