Speed breakers this week (a missing slash and incorrect pwd) – 1

This is first of series of posts I am going to write as and when it happens.

When trying to develop code, the first limitation is body and brain. (or heart and soul) ; But some times silly things take up all the available time wasting precious dev time. Here are two such things happened to me this week.

All I am trying to do is create a rudimentary web html page to try one of the REST API. (I know postman could be used, but I have plans to evolve this rudimentary into full blown web UI later – anyhow the REST API is already well tested with unit testing and postman)

 <form action="/restapi" method="post"> 

And it didn’t work!. Surprise, surprise!!
when I convert this to GET its working. I created so many forms in the past what the hell is happening? I check/recheck every bit by bit – deleted everything and retyped. Nothing is helping. There is no way this could go wrong. I double check, recheck my back-end code. Try different ways of doing back-end code. Is it because I am not including request.Form.Parse()? OK – no harm, let me add it. No luck. Took a break, and restarted with a fresh mind.
Finally a light bulb💡 – instead of worrying about back-end code I thought may be browser is not sending the post data, but that is not the case 😞 – anyhow, I Google searched for GET is working and POST sending empty data. Finally some result to help me 😥. If we don’t put a slash in the end it will redirect without post data.

I wrote so many forms in the past but most of them are for PHP or c#; this one is for golang. In PHP I used to give complete file name something like restapi.php so redirection didn’t came into picture. c# thanks to visual studio it is taken care.

After adding a slash it started working! Silly!!

 <form action="/restapi/" method="post"> 

Docker PWD

The second speed breaker is when trying to run ./main in docker I got incorrect permissions.

I added chmod +x ==> no luck.

I have a RUN cd main followed by CMD [“./main”] in Dockerfile ;

Expected PWD when CMD is being executed to be main directory. But that is the not the case –

so ./main ==> I assumed it will in main directory executing main exe; but docker is trying to run main directory and getting no permissions issue!

After staring at the screen for a long time, I realized what is happening. (of course did many not so useful search and reads)

Hindsight – I should never name directory and other files with same name. It adds confusion to the error messages. Making almost error message useless.

Fixed this by making WORKDIR /app/main before CMD.

Docker notes …

Basic commands:

docker image

docker image ls
(displays list of images available on this machine)
(or simply docker images instead of docker image ls)

docker image ls -a
(displays list of all images. I think it includes deleted images? intermediate images?)

docker rmi ==> will remove the image.

docker container

docker container ls
(displays all containers currently running)
(or simply docker ls)

docker container ls -a
(displays all containers including those stopped, exited)
(Question – can we restart a container?)
(Question – can we delete an image and still container be running?)

docker rm ==> will remove the container.

Image Vs Container

This link is good.
An image is an inert, immutable, file that’s essentially a snapshot of a container. Images are created with the build command, and they’ll produce a container when started with run. Images are stored in a Docker registry such as registry.hub.docker.com. Because they can become quite large, images are designed to be composed of layers of other images, allowing a miminal amount of data to be sent when transferring images over the network.

‘docker history’ is your friend

docker history command is your friend. It gives complete history of all the layers an image is built on including the size of each layer.

docker inspect is second best friend

docker inspect is awesome! It gives everything you need to know for a container.

Digging inside docker container

  1. docker attach (is one way)
    To detach try Ctrl + P / Ctrl + Q
    If it doesn’t work – try to kill the process which is attached. (pkill -9 -f 'docker.*attach')

2. docker container exec
e.g.. docker container exec 76192e17722e cd bin && ls


docker commit

This is another cool command. It makes a new docker image from running container. Take it to new machine and start running; or use it for debugging, Use the image to start container with additional options.

docker inspect

Looks like this is similar to docker history for image ;

Helpful tips for debugging container issues

  1. docker container exec <run any command like ps, top, ls>
  2. docker container exec -it <id> bash
  3. docker container exec -it -u 0 <id> bash
  4. docker container logs << Must run after you start a container>>
  5. docker container stats << view memory/CPU usage>>
  6. docker container top
  7. docker cp
  8. docker diff
  9. docker export (Gets all files as tar)

cleanup

  1. docker container prune ==> removes all stopped containers.
  2. docker image prune
  3. docker system df
  4. docker system prune (My favorite)
  5. docker rm $(docker ps -aq -f status=exited)
  6. This link is good.


Another doc link from docker to understand more about images vs containers.

Solving Sudoku like humans do

.. is not difficult, but intriguing.

Usually solved with blind back tracking. But it felt soul-less. I want to solve like humans do.

Step 1: Solve each cell which can be guessed with 100% probability.

Step 2: Pick one of the available options and move to next.

Step 3: If nothing matches, back-track and change one of previous cells done in step 2. If reached start still nothing – no solution found.

The whole program is at https://github.com/chavakiranasdev/kichavaLibrary/blob/master/leet/hard/sudoku/Sudoku/Solution.cs

It is written and tested on Leetcode, so we ended up converting input to our object and reconverted back to required format.

CellState

/// <summary>
/// Represents state of single Cell. 
/// </summary>
internal enum CellState
{
    Empty, // Not yet filled. 

    InitialValue, // This value is already given 

    SolvedWithConfidence, // We found it with 100% probability. 

    InProgress, // Currently in progress. 
}

CellLocation

/// <summary>
/// Represents a single cell location. 
/// Used during backtracking to remember path
/// </summary>
internal class CellLocation
{
    public int Row { set; get; }
    public int Column { set; get; }
}

Cell

/// <summary>
/// Represents a cell 
/// </summary>
internal class Cell
{
    public CellState State { get; set; }
    public int CurrentValue { get; set; }

    // We tried this number & it is not fit here.
    // Used during backtracking to not re-use discarded values.
    public List<int> NotFitForThisCell { get; set; }

    public Cell()
    {
        State = CellState.Empty;
        NotFitForThisCell = new List<int>();
    }

    public void Reset()
    {
        CurrentValue = 0;
        State = CellState.Empty;
    }

    public void ClearNotFitForThis()
    {
        NotFitForThisCell.Clear();
    }
}

Find a match with 100% probability

// This checks if there is only one fit for this cell. 
// If it finds more than one fit, then we return zero.
private int GetBestFit(int row, int column)
{
    var gridKey = GetGridNumber(row, column);
    var availableInAllThree = rowAvailableList[row].Intersect(columnAvailableList[column]).Intersect(gridAvailableList[gridKey]);
    if (availableInAllThree.Count() == 1)
    {
        return availableInAllThree.First();
    }
    return 0;
}

Datastructures

// Constants 
// we can generally solve sudoku of any size. 
// So far, tested only for size 9
private int SizeOfSudoku { get; set; }

// Private variables. 
// Later, we create a board of given size (e.g.. 9X9)
private Cell[,] cellBoard;

// Next three dictionaries (hash's) represent each row, column, grid and still available values at any instance. 
// For example, once we initialize with input values, we remove them from corresponding row, column, grid. 
// Instead of three hashe's we could use 9X9 hashes to do same for each cell. That should be good for perf, for now this what I used.
Dictionary<int, List<int>> rowAvailableList = new Dictionary<int, List<int>>();
Dictionary<int, List<int>> columnAvailableList = new Dictionary<int, List<int>>();
Dictionary<int, List<int>> gridAvailableList = new Dictionary<int, List<int>>();

Constructor

public Solution(int sizeOfSudoku = 9)
{
    SizeOfSudoku = sizeOfSudoku;
    cellBoard = new Cell[SizeOfSudoku, SizeOfSudoku];
}

Solution

public void SolveSudoku(char[][] board)
{
    Console.WriteLine("Start...");
    InitializeAllAvailableLiistsWithEverything();
    // Initialize given char array into our cell array. 
    InitializeCellArray(board);

    // First round - find all those which doesn't need guessing.
    bool isAtleastOneBestFitFound;
    int currentRow;
    int currentColumn;
    do
    {
        isAtleastOneBestFitFound = false;
        // Solve each row. 
        currentRow = 0;
        while (currentRow < SizeOfSudoku)
        {
            currentColumn = 0;
            while (currentColumn < SizeOfSudoku)
            {
                if (cellBoard[currentRow, currentColumn].CurrentValue == 0)
                {
                    var bestFit = GetBestFit(currentRow, currentColumn);
                    if (bestFit != 0)
                    {
                        // We have a match. Update the cell with this value.
                        isAtleastOneBestFitFound = true;
                        UpdateCellValue(currentRow, currentColumn, bestFit, CellState.SolvedWithConfidence);
                    }
                }
                currentColumn++;
            }
            currentRow++;
        }
    } while (isAtleastOneBestFitFound);

    // Second round - Try to do best guess
    bool response = true;
    response = SolveByRow();
    if (response == false)
    {
        Console.WriteLine("Bug: Unable to solve given Sudoku");
    }

    // we should be done now. 
    // Just put final result into char array instead of cell object array
    for (int i = 0; i < SizeOfSudoku; i++)
    {
        for (int j = 0; j < SizeOfSudoku; j++)
        {
            board[i][j] = Convert.ToChar(cellBoard[i, j].CurrentValue + 48);
        }
    }
}

SovleByRow

 private bool SolveByRow()
    {
        bool response = true;
        if (!SolveGivenSubBlock(rowStart: 0, rowEnd: SizeOfSudoku - 1, columnStart: 0, columnEnd: SizeOfSudoku - 1))
        {
            response = false;
            Console.WriteLine("Bug? Unable to solve some of the grid");
        }
        return response;
    }

SolveGivenSubBlock

 private bool SolveGivenSubBlock(int rowStart, int rowEnd, int columnStart, int columnEnd)
    {
        bool response = true;
        for (int currentRow = rowStart; currentRow <= rowEnd; currentRow++)
        {
            for (int currentColumn = columnStart; currentColumn <= columnEnd; currentColumn++)
            {
                if (!SolveGivenCell(currentRow, currentColumn))
                {
                    response = false; // do not overwrite this value for next cell's success, and thus above if
                }
            }
        }
        return response;
    }

SolveGivenCell

  private bool SolveGivenCell(int currentRow, int currentColumn)
    {
        bool response = true;
        var currentCell = cellBoard[currentRow, currentColumn];
        if (currentCell.CurrentValue == 0)
        {
            // go back tracking. 
            // This will first attempt to find best guess if not found then back tracks.
            if (!BackTrackRealThisTime(currentRow, currentColumn))
            {
                response = false;
                Console.WriteLine($"Bug? Unable to solve {currentRow} :: {currentColumn}");
            }
        }
        return response;
    }

Backtracking

 private bool BackTrackRealThisTime(int currentRow, int currentColumn)
    {
        var stack = new Stack<CellLocation>();
        // clear not fit for this, if any from prevous attempts.
        cellBoard[currentRow, currentColumn].ClearNotFitForThis();
        stack.Push(new CellLocation() { Row = currentRow, Column = currentColumn });

        while (stack.Count != 0)
        {
            var currentTopOfStack = stack.Peek();
            var bestGuess = GetBestGuess(currentTopOfStack.Row, currentTopOfStack.Column);
            if (bestGuess != 0)
            {
                // We have a guess. Update the cell with this value.
                UpdateCellValue(currentTopOfStack.Row, currentTopOfStack.Column, bestGuess, CellState.InProgress);
                stack.Pop(); // Remove this from stack.
                continue;
            }
            var previousCellLocation = GetPreviousInProgressCellLocation(currentTopOfStack);
            // Push this cell onto stack after emptying it.;
            var previousCell = cellBoard[previousCellLocation.Row, previousCellLocation.Column];
            previousCell.NotFitForThisCell.Add(previousCell.CurrentValue);
            RemoveCellValue(previousCellLocation.Row, previousCellLocation.Column);
            // Now that we are moving further back, current cell's notfit is no longer valid.
            cellBoard[currentTopOfStack.Row, currentTopOfStack.Column].ClearNotFitForThis();
            stack.Push(previousCellLocation);
        }
        return true; // stack is empty
    }

Conclusion

I tried to be lazy and backtrack in random, without remembering path – but on my machine it works, leetcode always timed out. Forcing me to be a better 🙂

Get rest of the code from above github link.

Notes while becoming a Gopher (aka learning Go language)

  1. Functions returning multiple values is taken to next level.  
  2. Golang’s documentation to get started is very well designed and implemented. Other languages can learn a bit from here to not make their complex-language learning extra-complex.  
  3. I thought I have seen last of pointers for major use cases, alas, I am mistaken.  
  4. “for” is new ‘while’.  
  5. Switch is all powerful.  (Good. I still remember how I suffered with switch in Java6 days)
  6. Methods are c# extension methods. (Or the otherway)  
  7. Go is C.Redefined. (Or C.Redesigned or C.Improved)  
  8. Go is not OOPS but Go is OOPS (Or Go is only a bit of OOPS and lot of Procedural)  
  9. Copy is actually CopyTo as destination is first argument and source is second argument.  
  10. Defer will do finally’s job, in more human way. Defer + Recover is catch; panic is throw exception. But the documentation says they don’t have try catch – I am a bit confused. It feels panic-defer-recover are more human friendly design of try-catch-finally.
  11. I may miss c#’s $_ string interpolation. 
  12. Loved separate name for rune. Makes conversing in Unicode characters more straightforward. 

An idea to save Twitter from itself : InDoor

Twitter is great – for celebrities, media and folks inside media.

But for common-man twitter is chaos. Nothing much to offer for them other than keeping up to date which also not possible due to its infinite chaos. Even if one follows ten people, it is impossible to feel satisfied. It is always unfinished – forcing people to run away to other social networking sites or moving on with life.

One idea to save Twitter from this self destructing nature. InDoor

Allow people to make their own twitter.

  1. Anybody can create an indoor twitter.
  2. Example twitter.com/indoor/myFamily or twitter.com/indoor/myTeam
  3. Tweets inside indoor are for members only.
  4. Every member follows every other member by default this setting can be changed either globally or per member by moderator.
  5. One can unfollow any member.
  6. One can retweet public tweets to indoor only.
  7. Once on indoor there won’t be any confusion whole twitter interface remains same, except everything now applies to indoor account. Users can easily exit indoor and go to public twitter.

This should make twitter more useful to common members who are not celebrities. This will put twitter in direct competition with What’sapp for family/friend groups + slack/teams for office groups.

Once basic structure is there – we could go on innovating much more.

Ideas to be tracked similar to bugs, tasks …

Ideas though important, are cheap, execution is costly.

Software engineering made leaps and bounds with agile, scrum and other related innovations. But one thing didn’t get much attention is ideas. We are not tracking them similar to our other objects of a project management. They are discussed in corridors, scrum meetings, emails and chat-channels. But that’s it. It is either hit or miss. Not preserved for future. Like every cell in a human body regenerates in seven years, every team regenerates with new faces in two years. Ideas are lost to ether. Who knows one of those ideas disappeared in a blip may save the project in future?

Of course there are reasons for ideas not being tracked.

  1. They are cheap! For somebody with startup itch knows how easy it is to be flooded with ideas and how travail execution of those ideas is.
  2. Ideas are not rewarded during review cycle. May be for right reasons, ideas itself are not rewarded during review cycle. What is delivered is the only value considered for all those fat bonuses, stock options and promotions. This leads to engineers and managers alike ignore the importance of ideas.
  3. Once we start tracking ideas, it might become frenzy. If something is tracked, there will be un?-necessary attention to it.

To avoid “3” above – to stop team going frenzy we could call ideas as proposals. Now along with bugs, tasks, backlogs, features we better have a place for Proposals. In that way anybody in team could propose something and be discussed, tracked and logged for history.

Circle-Spiral-Galaxy approach for whiteboard to release

There are two kinds of people – who are good at bird’s eye view and those good at ant’s eye view.

Bird folks are good at coming up with ideas, sub-ideas, thinking of all the problems, dependencies in reaching there; Ant folks are good at executing stuff but one grain at a time, how big the granary might be.

Now, keep both these folks in one room for whiteboard discussion and see the fun.

BTW, there are always ambidextrous folks, who can wear both the hats and context switch.

Here is a framework you can use to proceed from idea to release.

Form a big circle, write everything you can think of. organize them in whatever way you feel is suitable. This is basically your laundry list. It can include big picture design diagrams, north pole. If every dependency can be resolved, if we have infinite resources, what is that we are trying to achieve? How are we going to achieve it?

Now cut this big circle into three circles. Phase One, Phase Two, Phase Three. To be cute we can call them Milky-way, Andromeda, PinWheel. These are our short-term, medium term, long term goals.

Now each circle is to be cut into spiral with three parts. Zero, One & Two; Or Sun, AlphaCentaury & Proxima; Each one is its own release to customers. Sun could be start of life – gather data with initial POCs, perform things without dependencies, Work with dependent teams to unblock AlphaCentaury work.

Repeat this in three spirals in three small circles. We are still not actually executing project, only moving our laundry list into these galaxies followed by solar systems.

After each release revisit this and improve the same.

etFlix Advertising revenue – an unofficial gray paper

If anything the current quarter results and subsequent free fall of stock price tell, etFlix may no longer increase either subscription price or subscription base, it is time for other revenue sources. Otherwise its future is doomed. Moreover the content it is creating originally is super mediocre. Reason?: creativity is not something which can be produced by centralized managements.  It is altogether a different ball game, all the money etFlix could dump may not be enough for a competitive rich content.

O! and before I forget to mention, couple of side-tracks.

  1. In spite of all the head start and overwhelmingly superior engineering team presence its user’s home page plus suggestions is flawed. Even after browsing for 10 minutes couldn’t find a decent flick to watch. They only show either already watched or not interested to watch videos. They should at least have an option to hide something. Which should hide from every recommendation except hidden pieces.
  2. Prime is eating away its eye-time. It might be time to have a partnership with Costco (similar to T-Mobile) ; but Costco doesn’t get monthly 100$ average like T-Mobile. Once advertising based subscription is available etFlix may go to Costco for a partnership. It might be win-win for both of them.

Back to advertising based etFlix brain storming.

There are Six players in this

  1. Subscribers.
  2. Advertisers.
  3. Intermediaries.
  4. Sales
  5. Engineers.
  6. Advertisements

Subscribers:

You have existing subscribers and new subscribers who might come if it is advertisement supported model.

Once new advertisement based subscription comes – is it going to be completely free? Or 50% less than current price? For anybody else it might be easier but for etFlix with all its current user base this is difficult choice. If given free, current users might feel alienated, might even become vocal anti-advocates, a dangerous outcome. 💡 Instead etFlix should start with six months free for all the users who newly joined in advertising-based-subscription model, including those who got converted from any of current subscription models. This should keep current users happy and bring in new users in flocks as six months is a good trial period. After six months the subscription could be 50% less than current basic subscription. Of course this six months free can not be for-ever. It should be only for one or two quarters. Then it can be brought down to half-a-month free. Otherwise users will simply go on creating logins with different credit cards 😉

💡 In the initial days – we might be bombarded with new conversion requests. The service may not be ready for all those users trying to convert. Like Gmail initial invitation only method – we needed a way to tackle this problem. Robinhood’s add-to-queue is better approach. In this way we could completely control geography vs number of subscribers in new model. We could balance advertisement revenue vs loss in new subscription. So that the balance sheet at the end of quarter doesn’t look that bad in short term. This also means before we open up to users need to be ready with good advertisers.

Advertisers, Sales & Intermediaries

These are bunch of important folks. One of my MBA friends mentioned sales & marketing is a solved problem. Hopefully she is correct 🙂

In-house sales team can do everything in their capacity to get more advertisers. But this is equally technical problem too. In exact words this is digital enablement problem. Same problem Google adverbs solved. Anybody can be either advertiser or anybody can be a middle-man advocating other small to medium scale players. The big players can always be handled by in-house teams. Small and medium players may not get much attention. 💡 Here is where tech comes into picture. Anybody should be able to place an advertisement. All you need is a credit card.

Engineers

Without these folks, we couldn’t build! This team needs to be top-notch, highly responsive, iterative, experienced, creative with motivated leadership team.

Advertisements

This is the most important piece of puzzle. If this can be done correctly we will have success otherwise it will be another failed project. Except Youtube nobody is adopting latest technical advantages to video advertisement.

Unlike Google, etFlix may not have much personalized data about a particular user. But it has good enough data about viewers to depend upon and customize advertisements.

Example :

  1. Region,
  2. Favorite categories.
  3. Presence of kids – number of kids, their age group.
  4. Favorite color, favorite music etc…
  5. Income range?
  6. Education range?
  7. People with similar taste.
  8. Active time, free time,
  9. Data from other third parties (legal of course)

But the advertisements itself is a big challenge. We don’t want them to be highly intrusive. Do we support skip? What is the impact of CDN caching?

Do we support text only? image only? video only? (all three is what I recommend)

How does pay works? Any extra charge if user “saves” an advertisement to view later? 💡 Obviously we don’t want users to click on it and leave our video watching. We want users to continue watching video and enjoy the same. So we need a better idea to make advertisement win-win-win for etFlix-advertisers-subscibers. Instead of click and open in new tab we could have two buttons “Save” which will save it under a tab and also sends an email to user. “Contact Me” this will enable advertisers contact user with special coupons, offers etc..

The other fact is where to keep these advertisments?

  1. Banners (either text or image of GIF (sans-audio) )
  2. Full screen video advertisements with skip option.
  3. For high width screens – like those sports advertisements we could have vertical banners come and go on the right side.
  4. 💡 Dynamic placement advertisements. We all know there are several placement advertisements inside all those original videos. But they are static (for example actors will still be eating same pizza in same pizza-shop even after 10 years, that pizza shop may now be defunct!) we could make these dynamic! So Somebody watching in USA might see coffee boards, somebody in UK/India may see Tea bill boards! This will be completely dynamic! This is a great challenge for engineering team and content creators.

That’s all for now.

Rules of Engagement for a software engineer

  1. If you are stuck on a single problem for more than three hours it is time to call help.
  2. You work for your manager not for company. Is it?
  3. Keep your ideas to yourself until you have more around them, like useful impact for team, reasonable confidence of execution, POC & estimates 
  4. Best part of day must be spent on very important and more impacting work 
  5. Never over promise – which inevitably leads to under delivery. Always under promise and over deliver. 
  6. Tech debt vs timely delivery – always let the later win. Nobody loves a delayed but perfect release. (It may not even be that perfect)
  7. 20% of your time should be spent on something your manager is not aware at this instance. Provided you always work 20% more time than average.
  8. Consistent schedule is preferable for your health and your team’s health. Remember Rome is not built in a day.
  9. Know and understand dependencies of the feature before you write submit first draft of plan.
  10. Know your red-flags, and feel free to jump ship before they eat you alive.
  11. Every morning and evening write personal logs (as simple as OneNote )
  12. Nobody likes a smart-goblin.
  13. Always go prepared to a meeting.