Monday, May 4, 2015

Doing Data Analysis and Data Science in Python with pandas

Ever wonder how you can best analyze data in python? Wondering how you can advance your skills beyond doing basic analysis in tools like MS Excel or SAS? Want to learn how to do the data analysis in python and pandas? Then read this post.

pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real worlddata analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. It is already well on its way toward this goal.

pandas is well suited for many different kinds of data:

Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet
Ordered and unordered (not necessarily fixed-frequency) time series data.
Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels
Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure

The two primary data structures of pandas, Series (1-dimensional) and DataFrame(2-dimensional), handle the vast majority of typical use cases in finance, statistics, social science, and many areas of engineering. For R users, DataFrame provides everything that R’s data.frame provides and much more. pandas is built on top ofNumPy and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.

Here are just a few of the things that pandas does well:

Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data
Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects
Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let , , etc. automatically align the data for you in computations
Powerful, flexible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data
Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into DataFrame objects
Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
Intuitive merging and joining data sets
Flexible reshaping and pivoting of data sets
Hierarchical labeling of axes (possible to have multiple labels per tick)
Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving / loading data from the ultrafast HDF5 format
Time series-specific functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc.

Many of these principles are here to address the shortcomings frequently experienced using other languages / scientific research environments. For data scientists, working with data is typically divided into multiple stages: munging and cleaning data, analyzing / modeling it, then organizing the results of the analysis into a form suitable for plotting or tabular display. pandas is the ideal tool for all of these tasks.

Some other notes

pandas is fast. Many of the low-level algorithmic bits have been extensively tweaked in Cython code. However, as with anything else generalization usually sacrifices performance. So if you focus on one feature for your application you may be able to create a faster specialized tool.
pandas is a dependency of statsmodels, making it an important part of the statistical computing ecosystem in Python.
pandas has been used extensively in production in financial applications.

Lessons for New pandas Users

01 - Lesson: - Importing libraries - Creating data sets - Creating data frames - Reading from CSV - Exporting to CSV - Finding maximums - Plotting data
02 - Lesson: - Reading from TXT - Exporting to TXT - Selecting top/bottom records - Descriptive statistics - Grouping/sorting data
03 - Lesson: - Creating functions - Reading from EXCEL - Exporting to EXCEL - Outliers - Lambda functions - Slice and dice data
04 - Lesson: - Adding/deleting columns - Index operations
05 - Lesson: - Stack/Unstack/Transpose functions
06 - Lesson: - GroupBy function
07 - Lesson: - Ways to calculate outliers
08 - Lesson: - Read from Microsoft SQL databases
09 - Lesson: - Export to CSV/EXCEL/TXT
10 - Lesson: - Converting between different kinds of formats
11 - Lesson: - Combining data from various sources

<Note> This documentation assumes general familiarity with NumPy. If you haven’t used NumPy much or at all, do invest some time in learning about NumPy first.

For more resources, please visit the main repository.

pandas Cookbook

The goal of this cookbook (by Julia Evans) is to give you some concrete examples for getting started with pandas. These are examples with real-world data, and all the bugs and weirdness that that entails.

Here are links to the v0.1 release. For an up-to-date table of contents, see thepandas-cookbook GitHub repository. To run the examples in this tutorial, you’ll need to clone the GitHub repository and get IPython Notebook running. See How to use this cookbook.

A quick tour of the IPython Notebook: Shows off IPython’s awesome tab completion and magic functions.
Chapter 1: Reading your data into pandas is pretty much the easiest thing. Even when the encoding is wrong!
Chapter 2: It’s not totally obvious how to select data from a pandas dataframe. Here we explain the basics (how to take slices and get columns)
Chapter 3: Here we get into serious slicing and dicing and learn how to filter dataframes in complicated ways, really fast.
Chapter 4: Groupby/aggregate is seriously my favorite thing about pandas and I use it all the time. You should probably read this.
Chapter 5: Here you get to find out if it’s cold in Montreal in the winter (spoiler: yes). Web scraping with pandas is fun! Here we combine dataframes.
Chapter 6: Strings with pandas are great. It has all these vectorized string operations and they’re the best. We will turn a bunch of strings containing “Snow” into vectors of numbers in a trice.
Chapter 7: Cleaning up messy data is never a joy, but with pandas it’s easier.
Chapter 8: Parsing Unix timestamps is confusing at first but it turns out to be really easy.

The book can be downloaded here

Some Awesome Resources

<Note> All of the content, text, links, references in this post are taken from thepandas documentation website

Links and references do not represent wholesale endorsements of the resource or the author. This is not a comprehensive list, so I welcome your suggestions and comments.

Best,

Ali Syed

I look forward to your comments below and @aaalee.

Saturday, May 2, 2015

Automate the Boring Stuff with Python

Learn to Code

(Skip to Table of Contents)

Everyone in the 21st century needs to learn to code, right? But not everyone needs to become a software engineer or computer scientist. Automate the Boring Stuff with Pythonis written for office workers, students, administrators, and anyone who uses a computer how to write small, practical programs to automate tasks on their computer.

Have a folder with thousands of files that need to be renamed?
Need to look through thousands of rows in an Excel spreadsheet looking for ones to update?
Have to scrape data off of several web pages?

Normally this would involve hours of tedious clicking and typing. But programming your computer to do it can save you a lot of time and effort.

Automate teaches total beginners with no programming experience the Python programming language. (Though Part 2 will be of interest to seasoned developers as well.) Python has a gentle learning curve yet is also used by professional software developers. You don't need to know all the complexities of algorithms and syntax, you just want to write basic programs to automate mundane computer tasks. In the process, even total beginners will learn to use Python to control their computers without having to learn complex information about computer science. This is a practical programming guide for the rest of us.

Automate the Boring Stuff with Python is released under a Creative Commons license. You can read it, in full, on this website. Programming is a skill that should be in the hands of everyone: No, Seriously, You Should Learn to Code..

You can also support the author by purchasing a book or ebook: Purchase a copy ofAutomate the Boring Stuff with Python.

Part 1 - The Basics of Python Programming

Part 2 - Automating Tasks

Al Sweigart is the author of Invent Your Own Computer Games with Python, Making Games with Python & Pygame, and Hacking Secret Ciphers with Python. His books are freely available under a Creative Commons license from http://inventwithpython.com.

I400/590 : Large-Scale Social Phenomena - Data Mining Demo

Introduction

For your mid-term hack-a-thons, you will be expected to quickly acquire, analyze and draw conclusion from some real-world datasets. The goal of this tutorial is to provide you with some tools that will hopefully enable you to spend less time debugging and more time generating and testing interesting ideas.

Here, I chose to focus on Python. It is beautiful language that is quickly developing an ecosystem of powerful and free scientific computing and data mining tools (e.g. the Homogenization of scientific computing, or why Python is steadily eating other languages' lunch). For this reason, as well as my own familiarity with it, I encourage (though certainly not require) you to use it for your mid-term hack-a-thons. From my own experience, getting comfortable with these tools will pay off in terms of making many future data analysis projects (including perhaps your final projects) easier & more enjoyable.

Hopefully you already have Python installed. If you are new to it, search around for good introductory tutorials -- I'd say it has a forgiving learning curve, comparatively speaking.

IPython

IPython is a kind of add-on for Python that brings several improvements, most importantly for us are its interactive, graphical notebooks which provide a great way to quickly develop and share code. A gallery of interesting notebooks is provided here, including:

An introduction to IPython notebooks
Replications of Nate Silver's results in predicting the 2012 presidential elections, along with other assignments from Harvard's CS109 Data Science course.
Open-content for self-directed learning in Data Science - great notebooks illustrating some basic concepts in machine learning (Linear Regression, Logistic Regression, Random Forests, K-Mean Clustering)
A set of notebooks for Psychological model fitting and testing

Libraries

In this demo, we will demonstrate the basic functionality of several useful toolboxes, including:

Numpy - a library for manipulating matrices and performing linear algebra operations (matrix products, computing covariance matrices, etc.). All the other libraries build on the data structures provided by numpy. Here's a basic tutorial.
Scipy - a library that builds on top of numpy and provides more advanced linear algebra routines, statistical tools, and sparse matrices (useful in many applications, such as text mining)
Matplotlib - plotting library, similar in style to that used by MATLAB. Here's the official gallery (click on a plot to see the code to generate it) and here's an IPython notebook with many good examples.
Scikit-learn (a.k.a. sklearn) - machine learning library, making it easy to do anything from PCA to training classifiers. Here's a very quick quick start, and here's some tutorials showing how to do common machine learning tasks.
Pandas - is a library for managing data sets. It really makes a breeze of difficult or tedious tasks (loading / processing / cleaning data). There is a 10 min intro video here, a tutorial here, another series of tutorials by Hernan Rojas.
Networkx - a library for working with networks. This includes computing graph theoretic measures, laying out and plotting graphs. A brief tutorial from the docs is here.
NLTK - the Natural Language Toolkit, can be very useful for text mining and language processing tasks. Actually, we won't demonstrate it during the in-class demo, but you can see its utility in tutorials such as this andthis, as well as the documentation on the NLTK site.

You can find more useful links at Python for data analysis: the landscape of tutorials.

Data

Even simple analysis can extract interesting results from good data, but nothing can make up for bad data. There's a few places to find potential data sets, including publicly-available data sources (Tableau software provides some data sets, and there is a directory of APIs and data sources at ProgrammableWeb).

Another common strategy is to scrape data from the web. There is an automatic tool to do so built by Kimono (which I haven't used but looks impressive). Python has several tools to do this (many discussed here). We will focus on the combination of mechanize (which simulates a browser to download HTML) and BeautifulSoup (which parses the downloaded HTML). There are some good tutorials showing how to use mechanize and Beautiful here and here.

Installation

There are two ways to install the aforementioned tools. The first is to use a Python distribution that already comes with all of these included, such as Anaconda.

The second option is to use the Python installer. Once you have Python installed, run the following on the command line:

pip install -U ipython numpy scipy matplotlib scikit-learn pandas networkx nltk 
pip install -U mechanize beautifulsoup4

You may want to run this as administrator to install them system-wide. If this options doesn't work for you, I recommend trying Anaconda.

Demo

Download demo.zip

Thursday, April 23, 2015

Now you can start every R tutorial for free @DataCamp

http://blog.datacamp.com/now-can-start-every-r-tutorial-free-datacamp/

data scientist

Monday, May 4, 2015

Doing Data Analysis and Data Science in Python with pandas

Lessons for New pandas Users

pandas Cookbook

Some Awesome Resources

Saturday, May 2, 2015

Automate the Boring Stuff with Python

Learn to Code

Table of Contents

I400/590 : Large-Scale Social Phenomena - Data Mining Demo

I400/590 : Large-Scale Social Phenomena - Data Mining Demo

Thursday, April 23, 2015

Now you can start every R tutorial for free @DataCamp

Thursday, April 16, 2015

Beautiful plotting in R_ A ggplot2

Big Data Processing

Training recommendation (Tutorials from PyCon 2014 – USA) and contest update