Stock Price Data Analysis - 2

Let’s start by loading the data into memory first. The data is per minute stock prices for the SNP500 bucket for the 10 days prior to 17 Dec. This means we have 390 x 10 so about 3900 rows per stock. This, however, is not absolutely true since some data has holes in it - i.e. some stocks are missing some minute values. This needs to be taken care of after loading in the data.

Interpretation of PCA Results

Say we already have the results from the last post. Before going into the minute tickers data, let’s take a look at how PCA works and why we need it. PCA stands for Principle Components Analysis. It is one of the methods for reducing the dimensions in a dataset. It works by working out in which dimensions does the data vary most, and realigning the data along those axes, hence reducing the dimensions.

Stock Price Data Analysis - 1

Introduction I recently was able to get my hands on raw stock prices data (open, high, low, close and volume) for the Standard And Poor’s 500 index (SNP500). The data was for each stock in the SNP500 bucket (actually 504 stocks since some of the stocks are listed twice for different types of shares issued) at two types of intervals: Daily (from each market open date since Jan 1 to Dec 15 of 2016) Per minute for each of the 390 minutes the market is open for the past 10 days since 17 Dec ‘16.

Machine Learning and ASER Pakistan Education DataSet

I was looking to experiment with some of the machine learning techniques I had been learning the past few months. I was particularly interested if I could glean any insights about the education system in Pakistan. Despite the efforts of the government, a large proportion of the nation is illiterate and the state of education delivery remains poor. I am of the opinion that a very focused data-driven approach would be a better method of resolving the crisis that the country faces than spending tons of money on the distribution of laptops.

Social Injustice and the Individual's Moral Obligations

Abdul Sattar Edhi was a great man. He was selfless, generous, and along with his wife managed to build the Edhi Foundation, one of the largest charities in the country. The foundation provides welfare homes, ambulances, orphanages, morgues, and adoption services free of cost to whoever needs it. In a country like Pakistan where social welfare is in wanting, the foundation serves as a blessing to many. But how far does an individual’s efforts go in trying to alleviate the social injustices seen and experienced by millions in places where welfare systems are next to none?

On the Integration of Immigrants in Society

Laïcité. French secularism as defined by law is at its heart the complete and total separation of the church and state (or to be more comprehensive, of religion and state). In a recent discussion with a French colleague, I descended into heated debate on France’s failure to integrate immigrants into its society - endangering not just France but the whole of continental Europe through the rise of disgruntled and exasperated immigrants with little opportunity to grow.

Victor Frankl: Suffering and Meaning

Viktor Frankl was an Austrian neurologist and psychiatrist as well as a Holocaust survivor. In his book - Man’s Search For Meaning, he writes about the psychology of the prisoners in the Holocaust concentration camps. I very recently read a passage on suffering (and what meaning people can derive from it) in Frankl’s amazing book. Perseverance in the face of suffering is a subject I have given significant thought to, and Frankl eloquently summarises some of the most important aspects of suffering and its meaning in life.

Violence and its Effects on Intergenerational Mobility

Up until a few days ago I was not fully aware of the implications of systematic discrimination and bias against a particular race or ethnicity on its future generations. A lot has been said about how assimiliation of people from races or ethnicities discriminated against in the past into the mainstream economy is an arduous task. I used to think of all of this as pointless. To me the idea that people would blame the current living conditions of their people on their history was absolutely ridiculous, given the time period between them and the historic discriminations which had occurred.

Child Watches, Child Learns

I was sitting on the upper deck of the tram on my way to a friend’s place earlier today. It was raining heavily and it was a pleasure to hear the pattering outside, but my attention was drawn towards the very front. A very agitated kid of around 18 was standing in front of his seat and seemed to be in a trance - he would glance at his watch, then peer outside the window and then look back at his watch over and over again.

OpenCV 360 Video Stabilisation

Now that we have a way to plot great circles on to equirectangular planes, how do we use that to detect the pitch and roll of the camera and consequently stabilise the video?

Python for Great Circle Projections - Part 2

This post will be a bit mathematically involved. Bear with me. Projections Let’s jump right into the math. The formula for projecting any point in latitude, longtitude to its (x,y) equivalent on a rectangular plane is: x = λcos(φ1) y = φ λ is the longitude; φ is the latitude; φ1 are the standard parallels (north and south of the equator) where the scale of the projection is true; x is the horizontal position along the map; y is the vertical position along the map.

Python for Great Circle Projections

For an internal project at my company, I was asked to write code to draw great circles projected on to an equirectangular plane. Why this is useful I will discuss a little later, but first, what exactly are great circles? great circle on Earth A great circle is the largest circle that can be drawn on any given sphere. Every circle in Euclidean 3-space is a great circle of exactly one sphere.

Password Recovery Best Practices

Most of what I will mention here has been taken from experience and what I have read elsewhere about password recovery implementations. It is possible that this is not the best way for your project. Always take the security of user data seriously. Password recovery is one of the core features in user accounts management. Users should be given the opportunity to reset their passwords without a real person intervening, and this procedure should be designed to be secure.

Event-Driven Design for DOM Manipulation

After one of the projects in my company had reached a point where refactor was not only recomended, but necessary to clean up the code and make it maintainable, I decided to use a different approach for DOM manipulation in the client facing web app. Since we were doing a major refactor (yes, it does exist!), I was given considerable leeway to choose a different way of dealing with the big problem of manipulating DOM elements.

Javascript Web Application Development Flow

Recently, I was asked a question about Javascript development: how are modules used in ES6 and then how do they magically become one .js file when deployed. Since I have been using a workflow which answers both of these, I am going to write it up here for anyone who wants to know. Big projects can be written in modular form in one of the EcmaScript standards (Actionscript, Javascript, Typescript etc) and then compiled into one file for web delivery.

Terrains using Three.js and SRTM Data

Let’s jump right in to three.js plane buffer geometry. Three.js offers a buffer geometry object which is a simple array of height maps, much like our binary file we used from the SRTM data. Compared to the normally used plane geometry, buffer geometry uses much less overhead and is significantly faster if the object does not need to be modified once initialised. To make things simple, I will assume that the terrain we need to generate is contained only within one binary hgt file, i.e., the terrain is within one latitude and longitude cell (this means that the bounding box of the terrain will be within one integer long and wide: N58.79 - N58.92, E006.22 - E006.33 is within one integer wide and long while N58.79 - N59.12, E006.22 - E007.13 will require more than one hgt file to process and will be needlessly complicated for this example).

Using SRTM Elevation Data in Javascript

1 arc second SRTM Data was made publicly available by NASA a few months ago. SRTM Data is collected by the Shuttle Radar Topography Mission. It sweeps the surface of the earth to find irregularities which form the height map. According to the mission website: To acquire topographic (elevation) data, the SRTM payload was outfitted with two radar antennas. One antenna was located in the shuttle’s payload bay, the other on the end of a 60-meter (200-foot) mast that extended from the payload pay once the Shuttle was in space.

WGS84 Projections

For our WebGL Augmented Reality App, we needed to preprocess latitude and longitude values into localised coordinates. Lat/lng values are great, but they are not very useful if you need to represent them inside a 3D space. It is far better to use [x,y,z] coordinates which are stored as floats natively and can be used directly without projecting them into another coordinate space. Of course, you are better off using lat,lng values if you are looking at very large distances on a globe.

Perl File Management

We needed a simple file upload, view and delete app for one of our projects. Since my company uses a lot of perl, I was tasked to write the web app in the language. I have had some experience working on Perl before for other in-house projects, but this was the first time I was working from scratch. We decided the first prototype should not have account management because the project was still in its very early stages and should only have the bare essential UI to aid in file management on our servers.

Cesium to Visualise NYC Carbon Footprint

Cesiumjs is a wonderfully simple and extensible library to work with. It is, in simplest terms, a 3D globe with imagery providers which can be used for nearly anything under the sun. I recently discovered that NYC publishes a lot of its data online, and I was dying to play around with that data to see what I could gather from the city data. I also had Cesium on hand, and open city data coupled with a versatile visualisation tool is a killer combination so I gave myself a few hours last Saturday afternoon to see what I could do.