20 Oct

AWS S3 file upload from client side

Last week I pushed a new feature on production which involve file upload on AWS S3. When you are uploading file to S3 from frontend there is always a risk of exposing your AWS secrets to user, so you have following options to avoid this risk

  1. Involve server as middleware and upload via server API, you will have more control over it
  2. Allow users to upload directly to S3 anonymously

I don’t like both of above methods, as involving middleware you are uploading file twice frontend → server, server → s3 bucket and I don’t really want server to do this heavy lifting of uploading files to S3 on user behalf, while allowing user anonymous upload you are giving a blank check to you S3 bucket i.e. anyone in the world can upload to your s3 bucket, this was also not acceptable to me as this could have very swear consequences. I wanted to have bit more control over uploads while not compromising security issues, so I used AWS Security Token Service to generate secrets (access_key, access_secret) which lived temporarily and have limited access as you define it.

Configure an IAM user on AWS console to generate temporary secrets

In order to generate temporary secrets you need following configurations on AWS console

  1. create a new IAM user
  2. create a policy to allow upload on S3 bucket
  3. create a role for you IAM user to assume Security Token Service

Watch this video for configurations

Now, you can generate AWS STS with this ruby code

aws_sts = AWS::STS::Client.new(
          access_key_id: ENV['AWS_ACCESS_KEY_ID'],
          secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
          role_arn: ENV['AWS_ROLE_ARN'],
          role_session_name: 'session_name',
          duration_seconds: 12*60*60)
// this will generate access_key, access_secret

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are credentials you download after creating user and AWS_ROLE_ARN is value which I copied at the end of video

Enable ACL uploading to S3

You will be able to successfully upload files with secrets generated with above code, Since we have used a separate user to generate STS then only this user will be able to access file. To make this file accessible by other user as well you must send ACL: 'public-read' in request body or header while uploading file to S3. you can read more about ACL

I hope this help you improve your file upload as well. If you need any help please do ping me on twitter @alihaider907.

25 Nov

Newbies Introduction to Ruby on Rails

Note: This tutorial is intended to newbies only who want to kick-start web development with Ruby on Rails. I will just share links to different resources and give tips based on my experience which may help you starting development with Ruby on Rails.

Installation and Prerequisites

Just like any other framework ruby on rails framework has its own development environment, before diving into installations of environment I would recommend if you have  planned to use Window operating system for ruby on rails development please read Rails Development on Windows. Seriously and switch to linux/unix based operating system. If you still insist to carry on with Windows go ahead with my Best wishes.

You can find complete installation guide of Ruby on Rails on GoRails website (there are other websites too). I will recommend installing RVM instead of rbenv and installing it from source.  GoRails guides also include installing Git, MySQL/Postgres complete the installation process. You might get errors while installation please do check your OS and version before starting installation process.

Ruby Language

Once you have set up your installations and environment start with ruby language basics I would recommend The Little Book of Ruby by Huw Collingbourne. Don’t memorise anything just practice basic syntax, data structures, condition statements, loops modules and Classes. Since we have already installed ruby as well as created a project of rails (in above GoRails guide) your can either go to project root and open a console or write your program in a .rb file and run it with ruby keyword as mentioned in book.

Ruby on Rails Framework

You are familiar with Ruby language basics now it is time to start getting your hands on Ruby on Rails. You will find numerous tutorial on getting started with Rails, but RUBY ON RAILS TUTORIAL by Michael Hartl is the most comprehensive guide for newbies. It covers details at granular level starting from Zero to Deployment it includes everything. There are other good resources as well like RailsCastsDriftingRuby, RailsTutorial, but if you are newbie I would highly recommend finishing Ruby on Rails Tutorial book by Micheal Hartl thoroughly first and then look elsewhere like RailsCasts (very good for topic specific tutorials) as it will make your Rails framework understanding in-depth.

Development Environment

Development environment contains your machine, tools, settings, configurations etc. Make your development environment as smooth as possible use .bashrc, tumx, terminal settings aliases to amplify your concentration. Have strong grip over terminal commands. Choose any suitable text editor I personally use Sublime Text in development and vim or emacs in production mode. Look at StackOverFlow’s survey for Most Popular Developer Environments by Occupation.

Be Open Source 

Ruby on Rails is an open source framework and it has very large open source community, as a beginner you will get lots of help from open source community in term of different gems, git, gist, stack overflow etc. Don’t forget to contribute back once you have a good grip.

Follow Tech Talks

Ruby and Ruby on Rails framework has large community of developers across the world. There are many conferences being held across the world on Ruby language and Ruby on Rails framework both. RubyConf and RailsConf are two of the most popular conferences in Ruby on Rails world. Do follow tech talks in these conferences particularly from speakers David Heinemeier, Sean Griffin, Mike Perham, Aaron Patterson and people who are at the forefront of Ruby language and Ruby on Rails framework development. Look at 11 TOP INTERNATIONAL RUBY ON RAILS CONFERENCES AND EVENTS 2017. Also one particularly article I really like every Rails developer to read is The Rails Doctrine by DHH.

Never Stop Learning

Once you have develop basics of ruby on rails I would recommend having some DevOps and Front end frameworks like ReactJS, VueJS  skills as well. Do explore topics like High Availability, Consistency, Scaling RDBS, NoSQL etc and  regularly read engineering blogs of tech giants like Google, Twitter, Microsoft and Amazon etc.

If you are newbie to Ruby on Rails development and want any suggestions or help do ping me on twitter @alihaider907.

Happy Learning 😛

25 Mar

Sidekiq dashboard customised for urdu language

Recently Mike Perham creator of SideKiq gem tweeted about support of Arabic and Farsi language in Sidekiq dashboard and call for Urdu language support.

I being a native Urdu speaker and open source enthusiast rushed to open a Pull Request for Urdu language support .  Now you can have SideKiq dashboard in Urdu language as well. All you need to do is add following lines in your config/initializers/sidekiq.rb file

require 'sidekiq/web'

module Sidekiq
  module WebHelpers
    def locale

Now your Sidekiq dashboard will look like thisSidekiq Urdu GUI

Happy Background Processing  😎

20 Jun

Rails faker gem customised for Pakistani locale

Ever since I am working in open source I always have had wished to contribute towards open source community and thank GOD today I have submitted my first ever pull request to Faker gem. I have added customised locale for Pakistani names, provinces, telephone numbers, postal codes etc.
It is successfully merged to the original repository. I am the happiest person at the moment YAHOOOOOOO

20 Sep

Let’s debug nginx, unicorn errors

This tutorial is particularly intended for nginx, unicorn and rails environment. But you can replace unicorn with any Rake web server i.e. puma, thin, passenger etc. which runs behind nginx since they all communicate with nginx through sock files and these sock files most of the time become root cause of errors.

Hold a mug of coffee/tea and let’s debug your configurations.

Before digging into configurations make sure your nginx and unicorn are running properly. For nginx run following command and check nginx process is running or not

ps aux | grep nginx

For unicorn

ps aux | grep unicorn

If either of nginx or unicorn not running, make them run and check if this was all you needed.

Lets now go through with errors

502 Bad Gateway

One of the most common problem in unicorn nginx configurations is 502 bad gateway. Followings are possible reasons of 502 bad gateway

Sock file path

Root cause of 502 bad gateway is no communication between nginx and unicorn through a shared socket which means nginx cannot find sock file on which unicorn is listening on, check your nginx configurations

upstream unicorn_server { 
server unix:/path/to/your/unicorn.sock; 

sock file path in upstream block should exactly match listen sock file path in your unicorn conf file

listen '/path/to/your/unicorn.sock', :backlog => 64

If this is different for your configurations, make them same and restart your nginx and unicorn then check error.

Buffer Size

nginx buffer size could be another reason of bad gateway. Open your nginx log with tail and check whether it’s a buffer size issue

tail -f /var/log/nginx/error.log

Reload your home page and see if you get

upstream sent too big header while reading response header from upstream client 

in your nginx log. If Yes then open your nginx conf /etc/nginx/nginx.conf (default path) and add following to in http block

proxy_buffer_size   128k;
proxy_buffers   4 256k;
proxy_busy_buffers_size   256k;

restart nginx and reload page and check error

Permission Denied

If you are getting following response


then probably it is a sock file permission denied issue. Root cause of this error is when nginx cannot read unicorn’s sock file (i.e. when your unicorn sock file is owned by a user who has root or higher permissions then the nginx user)

This could be either solved by changing permission of the sock file so that nginx can read it or increase the permissions of nginx so that it can read it (but this is a bad way). Best way is to create your sock file inside /tmp directory and point nginx to the sock file inside /tmp directory ( if you are on fedora then sock file should be in /var/run/ )

Restart nginx and unicorn and check

if your on centos then you can be victim of running nginx as httpd_t or unconfined_t follow Nginx + Rails + Unicorn Permission Error: ‘sudo nginx’ vs ‘sudo service nginx start’ for details.

If you still facing same error please comment below with your nginx and unicorn logs. I will surely reply at earliest.

my mug is finished … Happy Deployment 😉