Saturday, September 30, 2017

Here's the code I used and the results I got testing against a Redis instance running on localhost.

import time
import redis
from redis.client import StrictPipeline



class StreamingPipeline(StrictPipeline):
    def reset(self):
        super(StreamingPipeline, self).reset()
        self.connection = self.connection_pool.get_connection('MULTI',
                                                              self.shard_hint)
        self.commands = []

    def execute_command(self, *args, **kwargs):
        self.connection.send_command(*args)
        self.commands.append(args[0])

    def execute(self, raise_on_error=True):
        connection = self.connection
        return [self.parse_response(connection, command_name)
                for command_name in self.commands]



def run(pipe, type):
    for num in (10, 50, 100, 1000, 10000):
        start = time.time()
        for _ in xrange(num):
            pipe.get('foo')
        result = pipe.execute()
        run_time = time.time() - start
        print "Processed %d %s GETs in %f seconds" % (num, type, run_time)
        pipe.reset()


client = redis.StrictRedis()

def streaming():
    pipe = StreamingPipeline(client.connection_pool,
                             client.response_callbacks,
                             False,
                             None)
    run(pipe, 'STREAMED')


def buffered():
    pipe = client.pipeline(transaction=False)
    run(pipe, 'BUFFERED')


buffered()
streaming()

Results:

Processed 10 BUFFERED GETs in 0.001150 seconds
Processed 50 BUFFERED GETs in 0.000862 seconds
Processed 100 BUFFERED GETs in 0.001425 seconds
Processed 1000 BUFFERED GETs in 0.010493 seconds
Processed 10000 BUFFERED GETs in 0.086214 seconds
Processed 10 STREAMED GETs in 0.000219 seconds
Processed 50 STREAMED GETs in 0.000801 seconds
Processed 100 STREAMED GETs in 0.001714 seconds
Processed 1000 STREAMED GETs in 0.016336 seconds
Processed 10000 STREAMED GETs in 0.168507 seconds

Redis: ZREM

http://grokbase.com/t/gg/redis-db/12cshy430m/how-to-delete-multiple-keys-using-python-api-and-delete-and-zrem

Both zrem and del support *args so you can pass lists to them:

keys = ('a', 'b', 'c')
r.zrem('my_key', *keys)
r.delete(*keys)

you probably want to do this in batches of a few thousands at a time, and
you can use the python slicing syntax to sent just part of the list.
suppose keys is a list of millions of rows, we can do this:

step = 1000
for idx in xrange(0, len(keys), step):
chunk = keys[idx:idx+step]
r.zrem('my_key', *chunk)
r.delete('*chunk)

Redis: ZADD

https://github.com/andymccurdy/redis-py/issues/771

>>> import redis
>>> r = redis.StrictRedis()
>>> d = {'foo': 1.5, 'bar': 2.5}
>>> r.zadd('testing1', **d)
2
>>> d1 = r.zrange('testing1', 0, -1, withscores=True)
>>> d1
[('foo', 1.5), ('bar', 2.5)]
>>> r.zadd('testing2', **dict(d1))
2
>>> r.zrange('testing2', 0, -1, withscores=True)
[('foo', 1.5), ('bar', 2.5)]

http://redis-py.readthedocs.io/en/latest/_modules/redis/client.html

https://redis.io/commands/zrangebyscore

https://www.w3resource.com/redis/redis-zadd-key-score1-member1.php

def zaddWithTimeScore(self, keyname, key1):
""" Redis zadd that uses the current time for the score
"""
score1 = str(float(time.time()))

print("keyname: " + str(keyname) + " key1: " + str(key1) + " score1: " + str(score1))

dict = {key1: score1}

self.cache.zadd(keyname, **dict)

Wednesday, September 27, 2017

CENTOS Dropbox setup

https://www.digitalocean.com/community/tutorials/how-to-install-dropbox-client-as-a-service-on-centos-7

sudo curl -o /etc/init.d/dropbox https://gist.githubusercontent.com/thisismitch/6293d3f7f5fa37ca6eab/raw/2b326bf77368cbe5d01af21c623cd4dd75528c3d/dropbox
sudo curl -o /etc/systemd/system/dropbox.service https://gist.githubusercontent.com/thisismitch/6293d3f7f5fa37ca6eab/raw/99947e2ef986492fecbe1b7bfbaa303fefc42a62/dropbox.service
sudo chmod +x /etc/systemd/system/dropbox.service /etc/init.d/dropbox
sudo nano /etc/sysconfig/dropbox
sudo systemctl daemon-reload
sudo systemctl start dropbox
cd ~
curl -LO https://www.dropbox.com/download?dl=packages/dropbox.py
chmod +x ~/dropbox.py
ln -s /opt/dropbox ~/.dropbox-dist
/opt/dropbox/dropboxd
/opt/dropbox/dropboxd
~/dropbox.py
~/dropbox.py status

SSH and port 22 article link

https://www.adayinthelifeof.nl/2012/03/12/why-putting-ssh-on-another-port-than-22-is-bad-idea/

Friday, September 22, 2017

CENTOS: installing redis

http://sharadchhetri.com/2014/10/04/install-redis-server-centos-7-rhel-7/

sudo yum install redis
systemctl start redis.service
sudo systemctl start redis.service
sudo systemctl status redis.service
redis-cli ping
sudo firewall-cmd --zone=public --add-port=6379/tcp --permanent
sudo firewall-cmd --reload

ss -nlp|grep redis

http://blog.andolasoft.com/2013/07/how-to-install-and-configure-redis-server-on-centosfedora-server.html

edit the /etc/redis.conf file and bind the servers ip address to the localhost address

bind 127.0.0.1 192.168.1.155

comment out the following line

bind 127.0.0.1

sudo systemctl restart redis

redis-cli -h 192.168.1.155 ping

Thursday, September 21, 2017

Opencv: object tracking resources

https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients

https://www.learnopencv.com/object-tracking-using-opencv-cpp-python/

http://www.pyimagesearch.com/2015/09/21/opencv-track-object-movement/

Linux: ss command: which service is listening

http://sharadchhetri.com/2014/09/27/ss-command-alternate-netstat/

ss command : alternate of netstat

September 27, 2014 by Sharad Chhetri Leave a Comment

In case you are looking for alternate of netstat command, here is called ss command.
By default ss command is shipped with package called iproute (Advanced IP routing and network device configuration tools)

Introduction of ss

ss command is another utility to investigate sockets. It is used to dump socket statistics. It allows showing information similar to netstat. It can display more TCP and state informations than other tools. It is considered as alternate of netstat command also.

Check ss version

Use the -V argument with ss command. It will show the version information
# ss -V
ss utility, iproute2-ss130716

Examples with ss command utility

Example 1. Display only listening sockets . Use argument -l

# ss -l

Example 2. Display all listening and non listening sockets . Use argument -a

# ss -a

Example 3. Display TCP sockets. Use argument -t or –tcp. Try with -l argument, it will print all listening TCP sockets.

ss -t

Sample:

[root@nix ~]# ss -t
State       Recv-Q Send-Q                                      Local Address:Port                                          Peer Address:Port   
ESTAB       0      0                                          192.168.56.101:ssh                                           192.168.56.1:38153   
[root@nix ~]# 
[root@nix ~]# ss -tl
State       Recv-Q Send-Q                                      Local Address:Port                                          Peer Address:Port   
LISTEN      0      100                                             127.0.0.1:smtp                                                     *:*       
LISTEN      0      128                                                     *:ssh                                                      *:*       
LISTEN      0      100                                                   ::1:smtp                                                    :::*       
LISTEN      0      128                                                    :::ssh                                                     :::*       
[root@nix ~]#

Example 4. Display UDP sockets. Use the argument -u or –udp . Try with -l argument

ss -u

Sample:

[root@nix ~]# ss -u
State       Recv-Q Send-Q                                      Local Address:Port                                          Peer Address:Port   
[root@nix ~]# 
[root@nix ~]# ss -ul
State       Recv-Q Send-Q                                      Local Address:Port                                          Peer Address:Port   
UNCONN      0      0                                                       *:43923                                                    *:*       
UNCONN      0      0                                                       *:39372                                                    *:*       
UNCONN      0      0                                                       *:bootpc                                                   *:*       
UNCONN      0      0                                                       *:bootpc                                                   *:*       
UNCONN      0      0                                                       *:mdns                                                     *:*       
UNCONN      0      0                                                       *:28977                                                    *:*       
UNCONN      0      0                                                      :::39372                                                   :::*       
UNCONN      0      0                                                      :::53064                                                   :::*       
[root@nix ~]#

Example 5. Show process using socket.Use argument -p or –processes. Try with other arguments

ss -p

Sample:

[root@nix ~]# ss -ltp
State       Recv-Q Send-Q                                      Local Address:Port                                          Peer Address:Port   
LISTEN      0      100                                             127.0.0.1:smtp                                                     *:*        users:(("master",1785,13))
LISTEN      0      128                                                     *:ssh                                                      *:*        users:(("sshd",1036,3))
LISTEN      0      100                                                   ::1:smtp                                                    :::*        users:(("master",1785,14))
LISTEN      0      128                                                    :::ssh                                                     :::*        users:(("sshd",1036,4))
[root@nix ~]#

Example 6. Show socket memory usage. Use argument -m or –memory

ss -m

Sample:

[root@nix ~]# ss -mta
State       Recv-Q Send-Q                                      Local Address:Port                                          Peer Address:Port   
LISTEN      0      100                                             127.0.0.1:smtp                                                     *:*       
  skmem:(r0,rb87380,t0,tb16384,f0,w0,o0,bl0)
LISTEN      0      128                                                     *:ssh                                                      *:*       
  skmem:(r0,rb87380,t0,tb16384,f0,w0,o0,bl0)
ESTAB       0      0                                          192.168.56.101:ssh                                           192.168.56.1:38153   
  skmem:(r0,rb87380,t0,tb23080,f4096,w0,o0,bl0)
LISTEN      0      100                                                   ::1:smtp                                                    :::*       
  skmem:(r0,rb87380,t0,tb16384,f0,w0,o0,bl0)
LISTEN      0      128                                                    :::ssh                                                     :::*       
  skmem:(r0,rb87380,t0,tb16384,f0,w0,o0,bl0)
[root@nix ~]#

The manual of ss utility is very helpful. To explore, use the command man ss for man pages. Or you use the command ss --help for getting help.

Reference of ss -help command

Free Samples
Print
Home
Family
Centos
Clouding

Considered
Copyrighted
Default
Diag
Free Samples

[root@nix ~]# ss -help
Usage: ss [ OPTIONS ]
       ss [ OPTIONS ] [ FILTER ]
   -h, --help  this message
   -V, --version output version information
   -n, --numeric don't resolve service names
   -r, --resolve       resolve host names
   -a, --all  display all sockets
   -l, --listening display listening sockets
   -o, --options       show timer information
   -e, --extended      show detailed socket information
   -m, --memory        show socket memory usage
   -p, --processes show process using socket
   -i, --info  show internal TCP information
   -s, --summary show socket usage summary
   -b, --bpf           show bpf filter socket information

   -4, --ipv4          display only IP version 4 sockets
   -6, --ipv6          display only IP version 6 sockets
   -0, --packet display PACKET sockets
   -t, --tcp  display only TCP sockets
   -u, --udp  display only UDP sockets
   -d, --dccp  display only DCCP sockets
   -w, --raw  display only RAW sockets
   -x, --unix  display only Unix domain sockets
   -f, --family=FAMILY display sockets of type FAMILY

   -A, --query=QUERY, --socket=QUERY
       QUERY := {all|inet|tcp|udp|raw|unix|packet|netlink}[,QUERY]

   -D, --diag=FILE     Dump raw information about TCP sockets to FILE
   -F, --filter=FILE   read filter information from FILE
       FILTER := [ state TCP-STATE ] [ EXPRESSION ]
[root@nix ~]#

###############################################

Redis Server listens by default at port number 6379. Use below given ss command. (To learn more about ss command)

[root@localhost ~]# ss -nlp|grep redis
tcp    LISTEN     0      128            127.0.0.1:6379                  *:*      users:(("redis-server",19706,4))
[root@localhost ~]#

programming matrix

Saturday, September 30, 2017

Good Redis-py example link

Redis-py: pipeline

Redis: ZREM

Redis: ZADD

Wednesday, September 27, 2017

CENTOS Dropbox setup

SSH and port 22 article link

Friday, September 22, 2017

CENTOS: installing redis

Thursday, September 21, 2017

Opencv: object tracking resources

Linux: ss command: which service is listening

ss command : alternate of netstat

Introduction of ss

Check ss version

Examples with ss command utility

Followers

Blog Archive