Thursday, December 28, 2017
Unresolved reference issue in PyCharm - reference
https://stackoverflow.com/questions/21236824/unresolved-reference-issue-in-pycharm
Open settings
https://www.jetbrains.com/help/pycharm/accessing-project-structure.html
Set content roots
https://www.jetbrains.com/help/pycharm/configuring-content-roots.html
Invalidate caches
https://stackoverflow.com/questions/11725519/pycharm-shows-unresolved-references-error-for-valid-code
Open settings
https://www.jetbrains.com/help/pycharm/accessing-project-structure.html
Set content roots
https://www.jetbrains.com/help/pycharm/configuring-content-roots.html
Invalidate caches
https://stackoverflow.com/questions/11725519/pycharm-shows-unresolved-references-error-for-valid-code
Monday, December 25, 2017
Emacs and pdb
M-x pdb
Sunday, December 24, 2017
Emacs, espy, and virtualenv's
There was a bug (see below) in my elpy/pdb .emacs configuration that showed up when I ran pdb in emacs. I must have missed something in the setup.
Here is the solution:
https://stackoverflow.com/questions/9167614/python-mode-in-emacs-no-such-file-or-directory-pdb
pdb problem
https://github.com/jorgenschaefer/elpy/issues/1001
Elpy really should find the correct python interpreter if it's available inside your virtualenv. The question only is whether M-: (executable-find "python") finds the correct executable, and if not, why not.
M-x elpy-config
http://blog.mijalko.com/2014/02/python-debug-in-virtualenv.html
http://davedash.com/tutorial/pdb-the-python-debugger/
|ImportError: No module named unidecode
Here is the solution:
https://stackoverflow.com/questions/9167614/python-mode-in-emacs-no-such-file-or-directory-pdb
Add this to your .emacs
(defvar virtual_env_dir)
(setq virtual_env_dir (substitute-env-vars "$WORKON_HOME"))
(pyvenv-activate virtual_env_dir)
(setq gud-pdb-command-name "python -m pdb")
(setq gud-pdb-command-name "python -m pdb")
File "process_scraped_records.py", line 1, in
| from datetime import datetime
|ImportError: No module named unidecode
class UrlType(Enum): |Uncaught exception. Entering post mortem debugging
MRP = 1 # multi url record page_type |Running 'cont' or 'step' will restart the program
https://github.com/jorgenschaefer/elpy/issues/1001
Elpy really should find the correct python interpreter if it's available inside your virtualenv. The question only is whether M-: (executable-find "python") finds the correct executable, and if not, why not.
M-x elpy-config
Elpy Configuration
Virtualenv........: kafka (/home/jsmith/python_virtual_env/kafka)
RPC Python........: 2.7.12 (/home/jsmith/python_virtual_env/kafka/bin/python)
Interactive Python: ipython (/home/jsmith/python_virtual_env/kafka/bin/ipython)
Emacs.............: 25.3.2
Elpy..............: 1.17.0
Jedi..............: 0.11.0 (0.11.1 available)
Rope..............: Not found (0.10.7 available)
Importmagic.......: 0.1.7
Autopep8..........: 1.3.3
Yapf..............: 0.20.0
Syntax checker....: flake8 (/home/jsmith/python_virtual_env/kafka/bin/flake8)
There is a newer version of Jedi available.
[run] python -m pip install --upgrade jedi
Options
Square brackets indicate buttons; type RET or click mouse-1 on a
button to invoke its action. Invoke [+] to expand a group, and [-] to
collapse an expanded group. Invoke the [Group], [Face], and [Option]
buttons below to edit that item in another window.
[+]-- [Group] Elpy
[+]-- [Group] Python
[+]-- [Group] Virtual Environments (Pyvenv)
[+]-- [Group] Completion (Company)
[+]-- [Group] Call Signatures (ElDoc)
[+]-- [Group] Inline Errors (Flymake)
[+]-- [Group] Snippets (YASnippet)
[+]-- [Group] Directory Grep (rgrep)
[+]-- [Group] Search as You Type (ido)
[ ]-- [Group] Django Extension
http://blog.mijalko.com/2014/02/python-debug-in-virtualenv.html
http://davedash.com/tutorial/pdb-the-python-debugger/
Virtualenv and project directory naming convention
From an elpy bug report:
A quick note on best practice here: Usually, you do create a virtualenv folder in a different place to your project directory (which will likely be in a VCS). For example, you can have a virtualenv in
~/.virtualenvs/my_project/ and the project directory in ~/projects/my_project.Thursday, December 21, 2017
Tuesday, December 19, 2017
Python date time
>>> from dateutil.parser import parse
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named dateutil.parser
https://stackoverflow.com/questions/20853474/importerror-no-module-named-dateutil-parser
On Ubuntu you may need to install the package manager
pip first:sudo apt-get install python-pip
Then install the
python-dateutil package with:pip install python-dateutilConverting strings to datetime
https://chrisalbon.com/python/strings_to_datetime.html
Python lib - basic dater times
https://docs.python.org/2/library/datetime.html
1 2 3 4 5 6 7 8 | >>> from datetime import datetime >>> from dateutil.parser import parse >>> import pandas as pd >>> war_start = '2017\/11\/27 13:03' >>> datetime.strptime(war_start, '%Y\/%m\/%d %H:%M') datetime.datetime(2017, 11, 27, 13, 3) |
Sunday, December 17, 2017
Saturday, December 16, 2017
Python ftp test
Python FTP
https://docs.python.org/3/library/ftplib.html
FTP test site
http://speedtest.tele2.net/
https://docs.python.org/3/library/ftplib.html
FTP test site
http://speedtest.tele2.net/
>>> from ftplib import FTP >>> ftp = FTP('speedtest.tele2.net') >>> ftp.login('anonymous') '230 Login successful.' >>> ftp.dir() -rw-r--r-- 1 0 0 1073741824000 Feb 19 2016 1000GB.zip -rw-r--r-- 1 0 0 107374182400 Feb 19 2016 100GB.zip -rw-r--r-- 1 0 0 102400 Feb 19 2016 100KB.zip -rw-r--r-- 1 0 0 104857600 Feb 19 2016 100MB.zip -rw-r--r-- 1 0 0 10737418240 Feb 19 2016 10GB.zip -rw-r--r-- 1 0 0 10485760 Feb 19 2016 10MB.zip -rw-r--r-- 1 0 0 1073741824 Feb 19 2016 1GB.zip -rw-r--r-- 1 0 0 1024 Feb 19 2016 1KB.zip -rw-r--r-- 1 0 0 1048576 Feb 19 2016 1MB.zip -rw-r--r-- 1 0 0 209715200 Feb 19 2016 200MB.zip -rw-r--r-- 1 0 0 20971520 Feb 19 2016 20MB.zip -rw-r--r-- 1 0 0 2097152 Feb 19 2016 2MB.zip -rw-r--r-- 1 0 0 3145728 Feb 19 2016 3MB.zip -rw-r--r-- 1 0 0 524288000 Feb 19 2016 500MB.zip -rw-r--r-- 1 0 0 52428800 Feb 19 2016 50MB.zip -rw-r--r-- 1 0 0 524288 Feb 19 2016 512KB.zip -rw-r--r-- 1 0 0 5242880 Feb 19 2016 5MB.zip drwxr-xr-x 2 105 108 12288 Dec 17 07:12 upload >>>
ProFTPD
sudo apt-get install xinetd telnetd
$ telnet localhost 21
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
UFW Firewall setup for ftp ports 20/21
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04
Just use webmin to setup ProFTPD
https://doxfer.webmin.com/Webmin/ProFTPD_Server
################################################
# Deprecated
################################################
Make an FTP user
useradd madmax -m -s /bin/bash
To change a users password you need to have sudo privileges yourself.
Make an ssh key for the user
Next let's convert it's key into RFC4716 format and put it into the proftpd folder:
ssh-keygen -e -f ~madmax/.ssh/id_rsa.pub | tee /etc/proftpd/authorized_keys/madmax
ProFTPD
http://www.proftpd.org/docs/howto/Debugging.html
ProFTPD installation with public keys
https://www.devops.zone/ftp-server/installing-proftpd-with-public-key-authentication-on-ubuntu-16-04/
As a security measure it is actually a good idea to comment the following line from /etc/ssh/sshd_config
and restart SSH:
sudo service ssh restart
Trouble shooting
http://www.proftpd.org/docs/faq/linked/faq-ch4.html#AEN258
sudo proftpd -nd10
http://www.proftpd.org/docs/contrib/mod_sftp.html
Limit directory access
http://www.proftpd.org/docs/howto/Chroot.html
Restricting Users' Directories
One of the most common questions for new users of ProFTPD is "How do I restrict my users to only certain directories?" or, phrased another way, "How can I put my users in a chroot jail?" As a common question, it definitely has a place in the FAQ. Many users, I fear, do not read the FAQ carefully, and so miss that section. The answer is ProFTPD's
This configuration directive may appear in the
Installing ProFTPd with Public Key Authentication on Ubuntu 16.04
$ sudo ssh-keygen -e -f ~ftp_user/.ssh/id_rsa.pub > t
$ sudo mv t /etc/proftpd/authorized_keys/ftp_user
$ sudo ls /etc/proftpd/authorized_keys/ftp_user
/etc/proftpd/authorized_keys/ftp_user
$ sudo service proftpd restart
https://extremeshok.com/4974/ubuntu-lts-ultimate-ftp-server-proftpd-mariadb-mysql-with-ftp-and-sftp-scp-quota-and-bandwidth-accounting/
Check SFTP installation
https://www.digitalocean.com/community/tutorials/how-to-configure-proftpd-to-use-sftp-instead-of-ftp
Use filezilla
Use the command line
Couldn't read packet: Connection reset by peer
https://unix.stackexchange.com/questions/274995/sftp-ssh-couldnt-read-packet-connection-reset-by-peer
$ telnet localhost 21
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
UFW Firewall setup for ftp ports 20/21
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04
Just use webmin to setup ProFTPD
https://doxfer.webmin.com/Webmin/ProFTPD_Server
################################################
# Deprecated
################################################
Make an FTP user
useradd madmax -m -s /bin/bash
To change a users password you need to have sudo privileges yourself.
sudo passwd username will change the password for the 'username' you want to change.Make an ssh key for the user
sudo -u madmax ssh-keygen -t rsa
Hit enter twice to confirm key creation without password.
Next let's convert it's key into RFC4716 format and put it into the proftpd folder:
ssh-keygen -e -f ~madmax/.ssh/id_rsa.pub | tee /etc/proftpd/authorized_keys/madmax
ProFTPD
http://www.proftpd.org/docs/howto/Debugging.html
ProFTPD installation with public keys
https://www.devops.zone/ftp-server/installing-proftpd-with-public-key-authentication-on-ubuntu-16-04/
As a security measure it is actually a good idea to comment the following line from /etc/ssh/sshd_config
Subsystem sftp /usr/lib/openssh/sftp-server
and restart SSH:
sudo service ssh restart
Trouble shooting
http://www.proftpd.org/docs/faq/linked/faq-ch4.html#AEN258
sudo proftpd -nd10
http://www.proftpd.org/docs/contrib/mod_sftp.html
Limit directory access
http://www.proftpd.org/docs/howto/Chroot.html
Restricting Users' Directories
One of the most common questions for new users of ProFTPD is "How do I restrict my users to only certain directories?" or, phrased another way, "How can I put my users in a chroot jail?" As a common question, it definitely has a place in the FAQ. Many users, I fear, do not read the FAQ carefully, and so miss that section. The answer is ProFTPD's
DefaultRoot configuration
directive, which accomplishes this functionality by using the
chroot(2) function.
This configuration directive may appear in the
proftpd.conf:DefaultRoot ~
Installing ProFTPd with Public Key Authentication on Ubuntu 16.04
https://www.digitalocean.com/community/tutorials/how-to-configure-proftpd-to-use-sftp-instead-of-ftp
https://www.devops.zone/ftp-server/installing-proftpd-with-public-key-authentication-on-ubuntu-16-04/
$ sudo ssh-keygen -e -f ~ftp_user/.ssh/id_rsa.pub > t
$ sudo mv t /etc/proftpd/authorized_keys/ftp_user
$ sudo ls /etc/proftpd/authorized_keys/ftp_user
/etc/proftpd/authorized_keys/ftp_user
$ sudo service proftpd restart
https://extremeshok.com/4974/ubuntu-lts-ultimate-ftp-server-proftpd-mariadb-mysql-with-ftp-and-sftp-scp-quota-and-bandwidth-accounting/
Check SFTP installation
https://www.digitalocean.com/community/tutorials/how-to-configure-proftpd-to-use-sftp-instead-of-ftp
Use filezilla
Use the command line
Couldn't read packet: Connection reset by peer
https://unix.stackexchange.com/questions/274995/sftp-ssh-couldnt-read-packet-connection-reset-by-peer
SFTP on Ubuntu
https://askubuntu.com/questions/420652/how-to-setup-a-restricted-sftp-server-on-ubuntu
Step 1 : Install OpenSSH package if not installed
Step 2 : Create separate group for SFTP users. Step 3 : Edit /etc/ssh/sshd_config file and make changes as below.
Find and comment below line.and add these lines to the end of the file. Step 4 : Restart sshd service. Step 5 : Add user with ftpaccess group and create password. Step 6 : Modify home directory permission. Step 7 : Create a directory inside home for upload and modify permission with group. That's it . Passwordless sftp http://www.adminschoice.com/how-to-configure-ssh-without-password |
Ubuntu webmin installation
https://www.digitalocean.com/community/tutorials/how-to-install-webmin-on-ubuntu-16-04
http://www.webmin.com/
http://www.webmin.com/deb.html
Can't login via root in the browser
https://stackoverflow.com/questions/29750502/unable-to-access-webmin-through-browser
UPDATE1: After installing Webmin again, I still can't access it through the browser. I tried the command:
http://www.webmin.com/
http://www.webmin.com/deb.html
Can't login via root in the browser
https://stackoverflow.com/questions/29750502/unable-to-access-webmin-through-browser
UPDATE1: After installing Webmin again, I still can't access it through the browser. I tried the command:
nc -zvw 3 10000
Connection to 10000 port [tcp/webmin] succeeded! $ sudo ufw allow 10000/tcp
Rule added
Rule added (v6)
$ sudo ufw status
Status: active
To Action From
-- ------ ----
20/tcp ALLOW Anywhere
21/tcp ALLOW Anywhere
22/tcp ALLOW Anywhere
10000 ALLOW Anywhere
10000/tcp ALLOW Anywhere
20/tcp (v6) ALLOW Anywhere (v6)
21/tcp (v6) ALLOW Anywhere (v6)
22/tcp (v6) ALLOW Anywhere (v6)
10000 (v6) ALLOW Anywhere (v6)
10000/tcp (v6) ALLOW Anywhere (v6)
$ sudo ufw disable
Firewall stopped and disabled on system startup
$ sudo ufw enable
Firewall is active and enabled on system startup
$ sudo service webmin restart Solutionhttp://akyl.net/how-reset-webmin-root-password $ sudo /usr/share/webmin/changepass.pl /etc/webmin root
Updated password of Webmin user root
Friday, December 15, 2017
Emacs elpy errors
https://github.com/jorgenschaefer/elpy/wiki/Installation
"ImportError: No module named jedi"
Either install jedi in your local virtualenv or globally.
"ImportError: No module named jedi"
Either install jedi in your local virtualenv or globally.
sudo pip install jedi
"ImportError: No module named epc.server"
Either install etc in your local virtualenv or globally.
Either install etc in your local virtualenv or globally.
sudo pip install epc
Datastax Python Cassandra tutorial
Wednesday, December 13, 2017
Monday, December 11, 2017
Scala - conditionally replace a column value - reference
https://stackoverflow.com/questions/32357774/scala-how-can-i-replace-value-in-dataframes-using-scala
1 2 3 | import org.apache.spark.sql.functions._ val newsdf = df.withColumn("timestamp", when(col("make") === "Tesla","S").otherwise(col("make"))) |
Emacs, Python, virtualenvwrapper, and Jedi references
https://github.com/tkf/emacs-jedi/issues/123
Read the virtualenvwrapper documentation for an alternative way of automatically activating project virtual evironments. In short you can create a
.dir-locals.el file in the root of your project, with the following content:((python-mode . ((project-venv-name . "myproject-env"))))
Change
"myproject-env" to the name of your virtualenv and activate the virtualenvironment using the python-mode hook:(add-hook 'python-mode-hook (lambda ()
(hack-local-variables)
(venv-workon project-venv-name)))
(add-hook 'python-mode-hook 'jedi:setup)
Subscribe to:
Posts (Atom)