Import and Match Multiple DataFrames from EXCEL using PYTHON PANDAS

Many times, there is need to concatenate and analyze multiple data frames in Python.

Suppose we have Three different Excel Sheets in which same type of attributes are there. From these excel sheets, the data can be imported to data frames and then analysis can be done.

Following are three different MS Excel Sheets of attendance of the candidates. From these sheets, we have to check whether a candidate attended all sessions or not

PYTHON CODE

import pandas as pd
df1 = pd.read_excel (r'Book1.xlsx')
df1 = pd.DataFrame(df1, columns= ['Name', 'Attended'])
df2 = pd.read_excel (r'Book2.xlsx')
df2 = pd.DataFrame(df2, columns= ['Attended'])
df3 = pd.read_excel (r'Book3.xlsx')
df3 = pd.DataFrame(df3, columns= ['Attended'])
print('---Sheet-1---')
print (df1)
print('---Sheet-2---')
print (df2)
print('---Sheet-3---')
print (df3)
frames = [df1, df2, df3]
result = pd.concat(frames, axis=1, sort=False)
print(result)

Output

Python Code: Synchronization and Race-Conditions with Multi-Threading

Thread synchronization: Mechanism to ensures that two or more concurrent threads do not simultaneously execute some particular program segment known as critical section

Critical section refers to the parts of the program where the shared resource is accessed.

Issues in Synchronization

Race Condition: Occurring of a condition when two or more threads can access shared data and then try to change its value at the same time. Due to this, the values of variables may be unpredictable and vary depending on the timings of context switches of the processes.

Python Code: Without Synchronization

import threading

x = 0

def increment_global():

   global x

   x += 1

def taskofThread():

   for _ in range(50000):

      increment_global()

def main():

   global x

   x = 0

   t1 = threading.Thread(target= taskofThread)

   t2 = threading.Thread(target= taskofThread)

   t1.start()

   t2.start()

   t1.join()

   t2.join()

if __name__ == "__main__":

   for i in range(5):

      main()

      print("x = {1} after Iteration {0}".format(i,x))

OUTPUT

x = 100000 after Iteration 0

x = 63883 after Iteration 1

x = 82552 after Iteration 2

x = 100000 after Iteration 3

x = 68994 after Iteration 4

Python Code: Locking Mechanism and Synchronization

import threading 

x = 0

def increment(): 

global x 

x += 1

def thread_task(lock): 

for _ in range(100000): 

lock.acquire() 

increment() 

lock.release() 

def main_task(): 

global x 

x = 0

lock = threading.Lock() 

t1 = threading.Thread(target=thread_task, args=(lock,)) 

t2 = threading.Thread(target=thread_task, args=(lock,)) 

t1.start() 

t2.start() 

t1.join() 

t2.join() 

if __name__ == "__main__": 

for i in range(10): 

main_task() 

print("Iteration {0}: x = {1}".format(i,x)) 

OUTPUT

Iteration 0: x = 200000

Iteration 1: x = 200000

Iteration 2: x = 200000

Iteration 3: x = 200000

Iteration 4: x = 200000

Iteration 5: x = 200000

Iteration 6: x = 200000

Iteration 7: x = 200000

Iteration 8: x = 200000

Iteration 9: x = 200000

Create EXE File from Python Code

Method 1: Generating .EXE using PyInstaller

Install PyInstaller in Anaconda Prompt. Then Execute the Python code with pyInstaller as follows:

Method 2: Generating .EXE using AUTO_PY_TO_EXE

Python Code: SHA: Secured Hash Algorithm

import hashlib

print ("Available Algorithms in HASHLIB: ", end ="")

print (hashlib.algorithms_guaranteed)

print ("\r")

str = "1"

result = hashlib.sha256(str.encode())

print("SHA256 Hash: ")

print(result.hexdigest())

print ("\r")

result = hashlib.sha384(str.encode())

print("SHA384 Hash: ")

print(result.hexdigest())

Python Code: MD5 Message Digest Algorithm for Encryption

import hashlib

x='mypassword'

result1 = hashlib.md5(x.encode())

print("Hash Generated : ", end ="")

print(result1.hexdigest())

Docker Platform

Virtualization is the use of virtual machine that can virtualize the greater part of the equipment assets, including processors, memory, and network availability. With the virtualization, physical equipment assets can be shared by at least one virtual machines. According to the necessities from Popek and Goldberg, there are three perspectives to fulfill the virtualization. To start with, the virtualization ought to give an identical domain to run a program contrasted with a local framework.

To comprehend virtualization, hypervisor ought to be addressed first. Hypervisor empowers correspondence amongst equipment and a virtual machine so that the virtualization fulfills with this deliberation layer (hypervisor). Hypervisor is originally called virtual machine screen (VMM) from these two terms (Hypervisor and VMM) are regularly regarded as equivalent words, yet according to the distinction from Agesen et al, a virtual machine screen (VMM) is a product that manages CPU, memory, I/O data exchange, interrupt, and the instruction set on a given virtualized environment.

A hypervisor may allude to an operating framework (OS) with the VMM. Commonly, a hypervisor can be partitioned into Type 1 and Type 2 hypervisor in view of the diverse level of execution. Sort 1 is sitting on equipment and the correspondence amongst equipment and virtual machine is immediate. The host operating framework is not required in Type 1 hypervisor since it runs straightforwardly on a physical machine. Because of this reason, it is now and again called an ‘exposed metal hypervisor’.

VMware vSphere/ESXi, Microsoft Windows Server 2012 Hyper-V, Citrix XenServer, Red Hat Enterprise Virtualization (RHEV) and open-source Kernel-based Virtual Machine (KVM) are distinguished in this class. Sort 2 hypervisor is on the operating framework to manage virtual machine effectively with the support of equipment arrangement from operating framework. The additional layer amongst equipment and virtual machine in the sort 2 hypervisors causes inefficiency contrasted with the sort 1 hypervisor. Virtual Box and VMware Workstation are in this class.

The terms of Host or Guest machine (or domain) are utilized as a part of the hypervisor to depict diverse parts. Host machine (domain) contains a hypervisor to manage virtual machines, and Guest machine (domain) implies each virtual machine sitting on a hosted machine in a protected and confined environment with its own consistent domain.

With these separated parts, the hypervisor can offer asset limits to different virtual machines on the same physical machine. In other words, the hypervisor is a product layer that makes a virtual domain with virtualized CPU, memory and I/O (stockpiling and network) gadgets by abstracting without end the underlying physical equipment. Virtual machine (VM) commonly alludes to an epitomized substance including the operating framework and the applications running in it also.

TYPES OF VIRTUALIZATION

· Full virtualization — Full virtualization gives virtualization without modifying guest operating framework. In x86 engineering, dealing with benefits instructions is a key component for virtualizing equipment. VMware offers binary interpretation of operating framework asks for so that virtualizing favored instructions can be finished without backings from either equipment or operating framework. There are Microsoft Virtual Server and VMware ESXi using this system.

· Para-virtualization- Xen gather initially created paravirt-operations (later called by paravirtualization) to bolster elite and effective asset segregation with slight adjustments to the guest operating framework. Xen saw that full virtualization upheld guest domains without an adjustment in the operating framework, yet there were negative influences on performance because of the utilization of shadow page tables. Paravirtualization (PV) requires the adjusted OS kernel with framework calls to manage advantaged instructions. Xen registers guest OS page tables specifically with the MMU with a read-just access to keep away from the overhead and intricacy regarding the updating shadow page tables in full virtualization. With the interface between a virtual machine and a hypervisor, paravirtualization accomplishes superior without the help from equipment expansions on x86. Paravirtualization underpins unmodified application binary interface (ABI) so that client applications don’t require any progressions. Paravirtualization is additionally called operating framework assisted virtualization in light of the familiarity with a hypervisor on guest OS. Xen, UML and VMware bolster paravirtualization.

· Hardware Assisted virtualization — To enhance performance of virtualization, Intel and AMD gives virtualization expansions to x86 processors. Intel Virtualization Technology (VT) and AMD Virtualization (AMD-v) are increasing velocities for favored instructions including memory management unit (mmu), coordinated I/O gadgets (iommu). With this equipment assisted virtualization innovation, altered guest OS is pointless to empower virtualization in light of the fact that VMM manages benefit instruction at a root mode which is a ring — 1 without affecting the guest OS.

Using Second Level Address Translation (SLAT), settled paging in Intel EPT (Extended Page Table) or AMD RVI (Rapid Virtualization Indexing), memory management has been improved and the overhead of translating guest physical addresses to genuine physical addresses has been decreased. Early CPUs for x86 don’t have virtualization expansions which are not included in equipment assisted virtualization.

· Operating System-level virtualization (Shared Kernel Virtualization) — Operating framework gives separated allotments to run virtual machines in a similar kernel. With a chroot operation, which is a move of a root catalog for a certain procedure with a detachment to outside registries, OS-level virtualization empowers segregation between numerous virtual machines on a mutual OS. Overhead is exceptionally restricted in this model because of the advantages of running under operating frameworks with a common kernel. Emulating gadgets or communicating with VMM is a bit much. The guest os and the host os ought to have a similar OS or kernel. Running Windows on Linux host is incompatible. There are Solaris Containers, BSD Jails, LXC, Linux vServer, and Docker.

SOFTWARE IMPLEMENTATIONS FOR VIRTUALIZATION

Bare-metal virtualization hypervisors

· Microsoft Hyper-V

· VMware ESX and ESXi

· Citrix XenServer

· Oracle VM

Hosted virtualization hypervisors

· VMware Workstation/Fusion/Player

· VMware Server

· Microsoft Virtual PC

· Oracle VM VirtualBox

· Red Hat Enterprise Virtualization

· Parallels Desktop

CONTAINERS

The approach of online programming, administrations and stages alongside commoditization of computing administrations implies that scaling has happened to paramount significance. Considerably more, new paradigms in programming advancement — lithe systems, which abbreviate the way between the engineer and the generation situations, additionally added to an increase in appropriation among ventures and new businesses alike.

These new advancement hones and another arrangement of necessities, as far as continuous integration and continuous sending meant that there was an appropriate seeding ground for thoughts like containerization. Containerization tools empower “unchanging nature” in the infrastructure: container applications that are worked ‘on the spot’ and ‘set up’ with an indistinguishable setup and same conditions from the original creators intended.

This is one of the main reasons (or maybe even the main driving power) of their fast selection and increased utilization. These tools permit application engineers to concentrate on the application instead of focusing on the infrastructure while bringing versioning to application picture dispersion. It even goes similarly as bringing ideas that “customarily” had a place with an alternate region: you can do pulls, pushes and confers on Docker pictures, ideas obtained from Source Code Management Software (like Git and Mercurial).

To get any disarray off the beaten path, Docker alludes both to an open-source extend (https://github.com/docker/docker) and additionally the organization behind it — Docker Inc. (once in the past dotCloud). dotCloud was a PaaS player who manufactured Docker for internal utilize. When they understood its potential, they rotated and concentrated only on developing Docker.

The significance of the venture in the more noteworthy biological system was likewise perceived by the vast majority of the industry, along these lines allowing Docker to bring 40M$ up in the latest funding round. Following the turn dotCloud (the PaaS offering of the organization) was “migrated” to Berlin-based cloudControl. Docker is only one of the accessible containerization offerings and, the same number of comparative ventures, depends on kernel highlights that (in Linux) have been accessible for over 6 years (since around 2007).

Notwithstanding, as with other innovations (see e.g. blast in cell phone deals with the introduction of iPhone) Docker has just put a client (and designer) neighborly interface around prior parts. Similar ideas showed up 2–3 years prior in Solaris OS (assemble 51 in Solaris 10), known as Solaris Containers w/Solaris Zones. While initially Docker was a wrapper for LXC (Linux Containers), these days it manages libcontainer — a brought together interface for cgroups and kernel namespaces.

DOCKER

While there is a huge amount of likenesses there are likewise no less than two tons of contrasts between the two. The most important distinction is that, in Docker, there’s no overhead introduced by the need to copy a whole OS for the virtual machine. Thusly, Docker gives better performance (as far as speed and limit) when for custom applications in a container, contrasted with virtual machines, gave that the VM host and LXC host have a similar equipment qualities.

Both containers and Virtual Machines address a similar issue — disconnection and control of parts of an application — however this is accomplished in various path as containers surrender a portion of the separation for a more effective use of the (host) framework assets.

DOCKER: AN INTRODUCTION

URL — https://www.docker.com

Docker depends for its execution surroundings, on elements in the host’s kernel — LXC. Be that as it may, it additionally needs filesystem bolster in the purported UFS (Union File System). Docker Filesystems Multilayer Early forms of Docker depended on AuFS, yet later discharges have received a “nonpartisan” approach, with an inclination for device mapper.

This utilization of a duplicate on-compose FS is really what makes Docker look like Git (and Docker Hub like GitHub). Docker’s two main ideas are Containers and Images, where Containers are, well, containers running the application and Images are the Containers-at-Rest (i.e. spared container state). To make a relationship with class ideas in OOP, Images are class definitions, while Containers are class instances at run-time.

With everything taken into account there are only 28 commands that the tool called Docker (self-entitled “independent runtime” for Linux containers) sees, yet they wrap all the specified capacities — including the control, (for example, begin/stop a container) and the meta-management, (for example, push/draw to/from the store).

For instance, starting a container running Redis is as basic as*:

$ docker pull mydockerfile/redis

$ docker run -d — name redis -p 6379:6379 mydockerfile/redis

This will allow one to subsequently use to connect to the Redis server.

<container_host_ip>:6379

The latest version of Ubuntu Linux (14.04 LTS) comes with activated Docker support. It still needs to be installed, though that’s just an ‘apt-get install docker.io’.

KEY FEATURES OF DOCKER

Application-driven: the incremental change that Docker brings on LXC is core interest on organization of use versus sending of machines

Compactness: once distributed, any picture will yield precisely the same wherever it runs • versioning: much like Git, permits submit/push and pulls on existing pictures, verifying contrasts from one focus on the other

Computerization: there are tools that permit a machine, once running, to achieve a particular state

Sharing: using Docker Hub anybody can expand on existing machines or make accessible their pictures for others

Reusability: a picture can be ‘fork’- ed for two distinct purposes.

INSTALLING DOCKER ON WINDOWS

The initial step is to download and install the most recent Docker for Windows work from its GitHub storehouse. As should be obvious in the following figure, Docker for Windows utilizes a conventional Windows installer.

In the event that you acknowledge the installation defaults (which I exceptionally recommend you do), the product gives you the following three parts:

Boot2Docker management tool and ISO: This is the Docker administration command-line interface (CLI).

Virtual Box: We require the center of the Oracle Virtual Box VM desktop virtualization item to permit us to run the center Linux code from inside Windows.

MSYS-git UNIX tools: The Docker Hub will pull containers from open Git storehouses, so we need customer tools accessible on our framework.

Next, double tap the Boot2Docker Start Desktop symbol to begin the Boot2Docker management tool. You’ll be provoked to permit Virtual Box to bind two virtual network interfaces, and after a short time you’ll see an unequivocally Linux-y command quick, as appeared in the following screenshot.

LAUNCHING TENSORFLOW

Open up “Docker Quickstart Terminal”. It ought to bring up a MinGW-sort shell. You could likewise utilize Windows cmd.exe or Powershell.exe however they have additional arrangements that you have to do before you can run docker.exe

Dispatch the TensorFlow container. Set up port forwarding, mount substance of registry into/home.

$ docker run — it — p 8888:8888 — p 6006:6006 — v/c/Users/name/Dropbox/mypath:/home b.gcr.io/tensorflow/tensorflow

The first run through the command is run, it will Download and install TensorFlow. A short time later, this ought to bring you into a Linux VM.

The command above mounts an organizer in your Windows host machine into the container. Its desirable over do things thusly, in light of the fact that the container does not hold on your documents as a matter of course.

It as of now accompanies Jupyter installed, yet you won’t have the capacity to get to the note pads by navigating to localhost:8888 (or whatever port Jupyter begins on). You will likewise want to open port 6006 to have the capacity to utilize TensorBoard (right now, exposing a port on a live container is unrealistic).

Instead, since it is running in a VM, not just do you have to forward the port (thus the — p 8888:8888), yet the IP address you get to should be the IP address of the VM, not the Windows Machine. Thus, you have to find the address of the docker machine running the container.

Listing the docker containers.

$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

444413d7235233 b.gcr.io/tensorflow/tensorflow "/bin/bash" 2 minutes ago Up 2 minutes 6006/tcp, 0.0.0.0:8888->8888/tcp MyPath

$ docker psdocker-machine ls

$ docker-machine ls

NAME ACTIVE DRIVER STATE URL SWARM ERRORS

default * virtualbox Running tcp://192.168.99.100:2376

$ docker-machine ip default

<My IP Address>

Navigate web browser to 192.168.99.100:8888 (or whatever webserver port your web app is running on) and you should be able to see your web apps.

TENSORFLOW AS A RESEARCH ENVIRONMENT

If you want to use this container for your research environment all you need to do is to clone the docker container, so that you can make changes to it.

In the Docker Quickstart terminal,

docker pull b.gcr.io/tensorflow/tensorflow-full

$ docker images

For example if you installed a python package like ipdb and want to make changes, type:

$ docker commit MyPath mypath/tensorflow

$ docker run mypath/tensorflow

Now, the container is ready for research purposes and implementation of algorithms.

This article focus on the assorted types of virtualization approaches, tools and technologies with the specific case of Docker Implementation for different applications. The concept of virtualization and its applications are quite immense and the more we dig into the matter, the more astonishing results we can fetch.

Free International Certifications during Lockdown

BitDegree is providing Free Certifications during lockdown in the domain of Digital Marketing.

The students, researchers, practitioners, academicians and industry experts can join these certifications in free of cost

https://www.bitdegree.org/tag/covid