# Home Welcome to my Knowledge Base! Welcome to **Ann's Knowledge Hub** - a place where I publish and share what have I learned through many years, and I am still learning as a software developer. This includes code snippets, debugging tricks, good reference articles, problems I faced and how I solved them, and my own how-to guides for technology. It also includes many key concepts I have learned, mainly on iOS platform, but also other stuff like Git, CI/CD etc. I am consolidating more and more content from various sources including my personal notes in Obsidian, physical notebooks, GitHub etc. ### Why I created this space I know this site will be a tiny drop in the sea of thousands of such sites and resources out there. But this is just an attempt to do my part in sharing what I have learned, hoping that someone will find this useful. At the very least, I come here and refer to many how-to guides regularly. Another reason to create this space was the inspiration from my two core principles of life - **Learn something new every day** and **Be the change you want to See.** I believe in continuous learning and sharing what I learn with others to help them along their journey. {% hint style="success" %} This knowledge hub is built using [GitBook](https://www.gitbook.com/about), a fantastic doc generator that provides rich content editing, easy edit/publish workflow and seamless integration with GitHub. I love it! :hearts: It is awesome and free for personal use. {% endhint %} # Bash My notes from reading the book 'Learn Bash the Hard Way' by Ian Miell. This is a great hands-on practical book that explains all the main concepts to work with bash. Highly recommended. ## Basic understanding of Command Line Appendix A: [Command Line Crash Course](https://learnpythonthehardway.org/book/appendixa.html) **1. Commands - Index cards** | Linux | | Windows | | ---------- | ----------------------------------------------------------------------------------- | --------------- | | `pwd` | print working directory | `pwd` | | `hostname` | my computer's network hostname | `hostname` | | `mkdir` | make a new directory | `mkdir` | | `cd` | change directory | `cd` | | `ls` | list directory | `ls` | | `rmdir` | remove directory | `rmdir` | | `pushd` | push directory | `pushd` | | `popd` | pop directory | `popd` | | `cp` | copy a file or directory | `cp` | | `mv` | move a file or directory | `mv` | | `rm` | remove a file | | | `less` | page through a file | `more` | | `cat` | print the whole file | `type` | | `xargs` | execute arguments | --- | | `find` | find files | `dir -r` | | `grep` | find things inside files | `select-string` | | `man` | read a manual page | `help` | | `apropos` | find what man page is appropriate | `helpctr` | | `env` |

look at your environment
prints all the environment variables

| --- | | `echo` | prints some arguments | `echo` | | `export` | export/set an environment variable | `set` | | `exit` | exit the shell | `exit` | | `sudo` | {DANGER} become super user | `runas` | | --- | run a command on many file | `forfiles` | | | | | | `ps` | Show all running static processes | | | `kill` |

Kills the process with a given id
kill -9 40458 violent kill

| | | | | | **2. Paths, folder, directories** ```sh $ pwd /Users/ann # MacOS /home/ann # Linux ``` **3. If you get lost** `pwd` tells you where you are. `cd ~` takes you home. ```sh $ pwd $ cd ~ ``` **4. Make a directory** `mkdir` creates a directory. `mkdir -p` creates directories that don't exist in the given path ```sh $ mkdir temp $ mkdir temp/apple/pear mkdir: temp/apple: No such file or directory $ mkdir -p temp/apple/pear # success ``` On Linux and macOS, `mkdir` with an existing directory will throw an error. If you have give `mkdir -p`, it stays silent. ```sh $ mkdir temp # Linux / MacOS $ mkdir teamp mkdir: cannot create directory ‘temp’: File exists $ mkdir -p temp $ # no error ``` If you give whitespace between the arguments of `mkdir`, it will create multiple directories. If you want to create a directory with spaces in it, enclose it in " " ```sh $ mkdir this is fun # creates 3 directories - this, is, fun $ mkdir "this is fun" # created 1 directory this is fun ``` **5. Change directory** `cd` changes directory. `cd -` takes you the previous directory you were in. ```sh $ pwd /Users/ann $ cd temp/apple/pear $ pwd /Users/ann/temp/apple/pear $ cd .. # ~/temp/apple $ cd ~ $ cd temp/something cd: no such file or directory: temp/something ``` If you give whitespace between arguments of `cd`, it would give error that there are too many arguments. If you want to navigate to a directory with whitespace, use `\` . Or enclose it in " " ```sh $ mkdir "this is fun" $ cd "this is fun" # navigates to the folder $ cd .. $ cd this\ is\ fun ``` **6. List directory ls** `ls` lists the contents of the directory ```sh $ pwd /Users/ann/dev/learn-bash/temp $ ls apple carrot $ ls apple orange pear ``` `ls -lR` shows the contents in recursive fashion. ```sh $pwd /Users/ann/dev/learn-bash/temp $ ls -lR total 0 drwxr-xr-x@ 4 ann staff 128 Mar 14 21:52 apple drwxr-xr-x@ 3 ann staff 96 Mar 14 21:52 carrot ./apple: total 0 drwxr-xr-x@ 2 ann staff 64 Mar 14 21:52 orange drwxr-xr-x@ 2 ann staff 64 Mar 14 19:08 pear ./apple/orange: total 0 ./apple/pear: total 0 ./carrot: total 0 drwxr-xr-x@ 2 ann staff 64 Mar 14 21:52 potato ./carrot/potato: total 0 ``` **7. Remove directory** `rmdir` deleted empty directory. If the directory is not empty, it throws an error. ```sh $pwd /Users/ann/dev/learn-bash/temp $ ls apple carrot $ cd apple $ rmdir pear $ ls apple orange $cd .. $ rmdir apple rmdir: apple: Directory not empty $ rmdir apple/orange $ rmdir apple # no error $ ls carrot ``` **8. Moving around - pushd, popd** `pushd` lets you go to a new location **after saving** your current location. It's like saying *remember where I am, then go here*.`popd` - you can return to your saved location. It's like saying, *the last directory I saved, pop it and take me there.* ```sh ➜ apple cd .. ➜ temp ll total 0 drwxr-xr-x@ 3 ann staff 96B Mar 14 22:13 apple drwxr-xr-x@ 3 ann staff 96B Mar 14 22:08 i ➜ temp$ pushd i/like/icecream ~/dev/learn-bash/temp/i/like/icecream ~/dev/learn-bash/temp ~/dev/learn-bash/temp/apple ➜ icecream$ popd ~/dev/learn-bash/temp ~/dev/learn-bash/temp/apple ➜ temp$ pwd /Users/ann/dev/learn-bash/temp ➜ temp$ pushd i/like ~/dev/learn-bash/temp/i/like ~/dev/learn-bash/temp ~/dev/learn-bash/temp/apple ➜ like$ pwd /Users/ann/dev/learn-bash/temp/i/like ➜ like$ pushd icecream ~/dev/learn-bash/temp/i/like/icecream ~/dev/learn-bash/temp/i/like ~/dev/learn-bash/temp ~/dev/learn-bash/temp/apple ➜ icecream$ pwd /Users/ann/dev/learn-bash/temp/i/like/icecream ➜ icecream$ popd ~/dev/learn-bash/temp/i/like ~/dev/learn-bash/temp ~/dev/learn-bash/temp/apple ➜ like$ pwd /Users/ann/dev/learn-bash/temp/i/like ➜ like$ popd ~/dev/learn-bash/temp ~/dev/learn-bash/temp/apple ➜ temp$ pwd /Users/ann/dev/learn-bash/temp ➜ temp$ pushd i/like/icecream ~/dev/learn-bash/temp/i/like/icecream ~/dev/learn-bash/temp ~/dev/learn-bash/temp/apple ➜ icecream$ pushd ~/dev/learn-bash/temp ~/dev/learn-bash/temp/i/like/icecream ~/dev/learn-bash/temp/apple ➜ temp$ pwd /Users/ann/dev/learn-bash/temp ➜ temp$ popd ~/dev/learn-bash/temp/i/like/icecream ~/dev/learn-bash/temp/apple ➜ icecream$ popd ~/dev/learn-bash/temp/apple ➜ apple$ popd popd: directory stack empty ➜ apple$ ``` If you run `pushd` without any arguments, then it will remember the current directory into the stack. Then you can go anywhere and do `popd`, it will take you to the previous directory. **9. Make empty files touchd** `touchd` makes an empty file with the given name. ```sh $ pwd ~/learn-bash $ touch me.txt $ ls me.txt ``` **10. Copy a file** ```sh $ cp me.txt you.txt $ ls me.txt you.txt $ mkdir yours $ cp you.txt yours/ $ ls me.txt yours ``` `cp -r` copies sub-directories with files in them. ```sh $ ls foo/ $ cp foo bar cp: foo is a directory (not copied). $ cp -r foo bar $ ls bar foo ``` **11. Moving a file (mv)** `mv` moves a file from location to another. It also renames a file. Works for files or directories: ```sh $ ls foo a.txt $ mv foo/a.txt foo/b.txt $ls foo b.txt $ mv foo foo-2 $ ls foo-2 ``` **12. View a file (less)** `less` displays the contents of a file page by page\ Use (spacebar) to scroll down and `w` to scroll up. To get out, press `q`. ```bash $ less myfile.txt # displays contents of the file ``` **13. Stream a file (cat)** `cat` just dumps the whole file to the terminal - with no paging or stopping. ```sh $ cat a.txt # file a contents $ cat b.txt # file b contents $ cat a.txt b.txt # file a contents file b contents ``` **14. Removing a file (rm)** `rm` deletes a file or directory. To remove recursively all the contents of a directory, use `rm -r` or `rm -rf` * `rm -f`: force deletion of file * `rm -r foo`: deletes the folder foo * `rm -i`: asks for confirmation to delete each file * `rm -r *`: deletes all folders in the current folder * `rm -r *.*`: deletes only the files directly in the current folder (not the inner folders or files) * `rm -R` works the same as `rm -r`\ Be careful when running recursive remove on files `rm -rf`. ```sh $ test ls a.txt b.txt $ test rm a.txt $ test ls b.txt $ cd .. $ ls foo bar test $ rmdir test rmdir: test: Directory not empty $ rm -r test $ ls foo bar ``` **15. Exiting your terminal** Use the `exit` command. ## Part 1 - Core Bash Page 1 - 34 ### What is Bash Bash is a shell program. **Shell program** is an executable binary (file that contains program instructions) that takes the commands that you type and when you hit return translates those commands into system calls to the OS API.\ Other shell programs - sh, csh, tcsh zsh .... You can run other shells from one another, eg: `tcsh` from `bash` *Bash* (1987) stands for *Bourne-Again*, the descendant of the Bourne shell (1977). *Zsh* originates in 1990. #### History of Bash !\[\[Pasted image 20250316222032.png|800]] **Exercises** 1. Run `sh` from a bash command line. What happens? Ans: It shows sh-3.2 and starts the `sh` shell. 2. Which commands does NOT work in `sh` - Examples: `ll`, `la` do not work. But `ls`, `echo`, `mkdir`, `touch` etc. work. ### Globbing and Quoting #### Globbing `ls *` - lists the files in the folder matching \* , i.e. all files`echo *` also lists all files in the folder`*` stands for 'all' - it is one of the **globbing** primitives. #### Quoting You can enclose `*` in single or double quotes to escape it. ```sh $ ls '*' ls: *:No such file or directory $ echo '*' * ``` #### Other glob characters * `?` - matches single character * `[abd]` - matches any character from a, b or d * `[a-d]` - matches any character from a, b, c, or d ```sh $ ls a ab abc b bb cc dd file1 file2 file3 $ ls [ab] a b $ ls [ab]b ab bb $ ls [ab]* a ab abc b bb $ ls file[0-9] file1 file2 file3 $ [a-c]* a ab abc b bb cc ``` #### Dotfiles Their names starts with a dot `.`. They are hidden files and don't show up with `ls`. You must use `la` to see them. ```sh $ mkdir .hidden $ touch .hidden/.a.txt $ ls # does not show .hidden folder $ ls .hidden # empty output $ la .hidden $ la .hidden .a.txt ``` Single dot folder - `.` is a file that represents the current folder. Double dot folder `..` is a special folder that represents the parent folder. #### Differences to regular expressions Globs look like regular expressions, but are used in different contexts.`*` character in the context of regex is used a suffix to `.`, eg: `rename -n '-s/(.*)/new$1/'` renames all files to have the prefix `new`\ since the `*` character is enclosed in quotes, it is NOT interpreted by the shell. #### Exercises * [x] Create a folder with files with similar names and use globs to list them * [x] Find how `grep` works ### Variables *(2025-03-16 11pm, Faircrest home)* #### Basic variables You create a variable by giving it a name, followed by `=` and then the value of the variable. eg: `FOO=bar`. You can see the value of the variable using `echo` and `$` then the variable name.\ Variable names are capitalized by convention, but not required. You can use quotes `"` to group multiple words into the value ```sh $ FOO=bar $ echo $FOO bar $ MULTI="two words" $ echo $MULTI two words ``` #### Quoting variables You can embed variables in each other. However, if you enclose them in double-quotes `" "`, bash will translate the variable into its value. If you enclose it is single quotes `' '`, it will NOT translate it. ```sh $ OUTER="this has $FOO in it" $ echo $OUTER this has bar in it $ OUTER_SINGLE='this has $FOO in it' $ echo $OUTER_SINGLE this has $FOO in it ``` #### Quoting and globs This does not happen with globbing - globs (eg: `*`) in variables are not expanded when in single or double quotes. #### Shell variables These are shell variables that are set up when bash starts, eg `PPID` is bash's parent process. It is a readonly variable. ```sh $ echo $PPID 50599 $ PPID=nonsense zsh: read-only variable: PPID ``` Variables made with the `export` command persist across bash restarts. The new bash session inherits the value set in the previous session.\ Note - this happens only in the bash sessions started by the running shell. So if you open a new terminal and type that variable, it will be empty. ```sh # regular variable $ FOO=bar $ echo FOO bar $ bash $ echo $FOO # empty # shell variable $ export NEW_FOO=bar $ echo $NEW_FOO bar $ bash $ echo $NEW_FOO bar # value persists ``` #### env and export commands If you want to see all the variables exported to processes that you start in the shell, use `env` command. #### Simple arrays Bash treats array in a special way. You can access arrays only by its index, that too encloses in . Example - `BASH_VERSINFO` returns an array with 6 elements (0 - 5). ```sh $ bash --version GNU bash, version 3.2.57(1)-release (arm64-apple-darwin24) Copyright (C) 2007 Free Software Foundation, Inc. bash-3.2$ echo $BASH_VERSINFO 3 bash-3.2$ echo ${BASH_VERSINFO[0]} 3 bash-3.2$ echo ${BASH_VERSINFO[1]} 2 bash-3.2$ echo ${BASH_VERSINFO[2]} 57 bash-3.2$ echo ${BASH_VERSINFO[3]} 1 bash-3.2$ echo ${BASH_VERSINFO[4]} release bash-3.2$ echo ${BASH_VERSINFO[5]} arm64-apple-darwin24 bash-3.2$ echo ${BASH_VERSINFO[6]} # empty ``` ### Functions You can have functions in bash. There is no type checking, it just works.\ Functions can have 'local' variables that are viewed and accessed from within the function. Variables defined outside can be accessed inside too. `local` can be used only inside a function. ```sh $ function foo { > echo Hello world > } $ foo Hello world $ function foo { > echo $1 > echo $2 > } $ echo "Hello Ann" Hello Ann $ echo Hello Ann Hello Ann ``` ```sh $ function bar { > local loc="hello" > echo $loc > } $ bar hello $ local loc="ddd" bash: local: can only be used in a function ``` Commands can be one of these - *builtins*, *functions*, *programs* or *aliases*. #### Builtins Commands that come out of the box in bash. eg: `builtin pwd`. * there is a special `type` builtin which tells how a command will be interpreted by the shell. eg: `type ls` returns `ls is hashed (/bin/ls)`, `type cd` => `cd is a shell builtin` * `source` is also a builtin**Programs** - executable files. You can see where the binary is by typing `which cd` returns `/usr/bin/cd`**Aliases** - alternate names given to commands. ```sh $ alias cd=something $ cd nothing: command not found $ unalias cd $ cd # ok now ``` ### Pipes and redirects #### Basic redirects Redirect takes the output from the preceding command and sends it to a file you specify. The redirect operator is `>`. ```sh $ echo "simple content" > a.txt $ cat a.txt simple content ``` #### Basic pipes Pipe takes output of one command and passes it as input to another command. `|` is the pipe operator. ```sh $ echo "line one > line two > line three > line two again > line three again" > b.txt $ cat b.txt | grep two line two line two again ``` #### File descriptors In Unix, everything is a file - when you push/pull data to/from things - whether you write to the terminal or an actual file or network interface. This gives a common interface to send data into - this is a *sink*, an entity that you can *push data into*.\ If you want to write to a terminal, you 'open' the file that corresponds to the terminal and get back a **file descriptor**. It is a number associated with the process that represents that file. You can write to that file descriptor and the OS will send the data to the right place. Each process gets three file descriptors by default: * `0` is *standard input* * `1` is *standard output* which is linked to the terminal by default. So the output of all commands eg: `cat b.txt` is sent to the terminal and you see it there. * `2` is *standard error* which is also linked to the terminal by default. So the output of all commands eg: `cat some.txt` which is an error is sent to the terminal and you see it there. The pipe operator `|` sends *only the stdout data* of the left command to the right command. It does *NOT capture data sent to the stderr*. * So if the left command succeeds, then the data in stdout is passed onto the right command * if the left command fails, then the data is in stderr, not stdout. So it is NOT passed to the right command. Instead it is just displayed in the default descriptor of stderr, which is the terminal.\ The same applies to the redirect operator `>`. It send only the stdout data to the file. stderr data is not sent to the file. If you want a specific output to a specific sink you can use the `n>` syntax where `n` is the file descriptor id.\ Let's understand this by running a command that does not exist, eg: `blah` and throws an error.\ By default, stderr is sent to the terminal, so you see it in the terminal.\ If you redirect it to a file, then nothing will be written in the file (because > does not send stderr data to the file) ```sh $ blah bash: blah: command not found $ blah > err.txt bash: blah: command not found $ cat err.txt # file is empty ``` If you want to send the error to a file, then use `2>` as the redirect option. Remember `2` is the file descriptor for standard error.\ If you want to ignore the error (i.e not show in the terminal), then redirect it to the 'black hole' file `/dev/null`. ```sh $ blah 2> err.txt $ cat err.txt bash: blah: command not found $ blah 2> /dev/null # no error displayed here ``` If you use `2>&1` it means stderror data (`2` is stderror) must go to wherever stdout was going at that time (`1` is stdout)\ Note: *the order of the redirect operator matters.* ```sh # by default stdout goes to terminal $ blah 2>&1 > aa.txt bash: blah: command not found # the error is displayed in the terminal because stdout was set to terminal by default. $ aa.txt # empty because the error was shown in the terminal # now let's flip the order of the commands $ blah > bb.txt > 2>&1 # here, we are first setting the stodout to go to bb.txt, then setting stderr to go where stdout goes (so it goes to bb.txt) # no error in the terminal because stderr went to where stdout was set (in this case bb.txt) $ cat bb.txt bash: blah: command not found ``` #### Standard out vs. standard error * *pipe* operator passes standard *output* of one command as the standard *input* of the next command * *redirect* operator sends file descriptor (could be stdout or stderr as we saw above) to a file. The following three commands produce the same result. ```sh # set up the file $ echo "line one > line two > line three > line two again" > a.txt $ grep two a.txt line two line two again $ cat a.txt | grep two line two line two again $ grep two < a.txt line two line two again ``` Double redirect `>>` appends the text to the end of the file. Single `>` overwrites the file if it exists. ```sh # use > operator (overwrites the file) $ echo hello > c.txt $ cat c.txt hello $ echo world > c.txt $ cat c.txt world # use >> operator (appends to the file) $ echo hello > c.txt $ echo world >> c.txt $ cat c.txt hello world ``` ### Basic bash scripting **Shell script** is a collection of shell commands that can be run non-interactively.\ Note - in the command below, you must use single quotes `' '` instead of double quotes `" "`, otherwise you get the error "bash: !/bin/bash": event not found" ```sh $ echo '#!/bin/bash' > my_script $ echo 'echo I am a script' >> my_script $ ls my_script $ ./my_script bash: ./my_script: Permission denied ``` The script failed because it is not marked as an executable. To correct this, use `chmod` to give exec permission to the script. ```sh $ chmod +x my_script $ ./my_script I am a script ``` If you run the script as `my_script` without the prefix `./`, it would fail because the current directory is not in the `PATH` variable. (this is where bash looks through to find commands) ```sh $ my_script bash: my_script: command not found $ echo $PATH /usr/sbin:/usr/bin $ PATH=${PATH}:. $ echo $PATH /usr/sbin:/usr/bin:. $ my_script I am a script ``` #### Startup scripts When bash starts up, it runs a series of files to set up the environment that you see at the terminal. It follows a sequence of paths depending on various conditions * **normal** vs.\*\* **remote**. **normal** - you start with `bash` command - the default terminal. **remote** - running under `ssh / rsh` * **login vs. non-login**. login - you started by logging in with username/password/keys/ non-login - you started with no credentials * **interactive vs. non-interactive**. interactive: you can type things in. non-interactive: running as a script * Files loaded in **bash** * /etc/profile, /ets/bash.bashrc, \~/.bash\_profile, \~/.bash\_login, \~/.bashrc, Environment variable $BASH\_ENV, runs bash, then at the end \~/.bash\_logout * Files loaded in **zsh** * /etc/zshenv, \~/.zshenv, \~/.zprofile, \~/.zshrc, \~/.zlogin, runs bash, then at the end of session \~/.zlogout If you want to avoid any startup scripts from running, do `env -i bash --noprofile --norc` * `--noprofile` asks bash not to source system-wide startup files * `--norc` asks bash not to source the user-specific startup files #### source builtin `source` runs the script within the context of the current shell. So, any variables defined in the current shell are available inside the script. If you don't use `source`, then those variables will not be available to the script. ```sh $ FOO=bar $ echo 'echo $FOO' > scr.sh $ chmod +x scr.sh $ ./scr.sh $ # empty because the variable FOO is not available inside the script $ source ./src.sh bar # now the variable FOO is available ``` ## Part 2 - Scripting Bash Page 35 - 73 #### Command substitution It allows you to place the standard output of one commands into the script as you manually wrote it there. Use `$( )` or the backquotes for this.\ Note - if you use single quotes `' '`, it will ignore the meaning of `$` and will print it verbatim. So use double quotes `" "`\ It is better to use the `$( )` because the backtick option requires backticks to be escaped when you want to nest them ```sh $ hostname Mac.lan $ echo 'My hostname is $(hostname)' My hostname is $(hostname) # use double-quotes - with $() or `` $ echo "My hostname is $(hostname)" My hostname is Mac.lan $ echo "My hostname is `hostname`" My hostname is Mac.lan ``` #### Exit codes A variable that tells you the result of a command (or function or built-in). It is like the HTTP error code.\ You can get the exit code using the command `$?`. | Exit code | Meaning | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | OK. No Error | | 1 |

Generic Error (used when there is no specific error code for the command).
A grep command that does not see matching text has exit code 1.

| | 126 | Cannot execute due to permission issue | | 127 | Command not found (no file found for the command) | | 128+n | Process killed with a signal 'n'. Eg: 130 = process killed with Ctrl-C (signal 2) | | 3 - 125 | not reserved. Available for everyone. Use `exit` builtin followed by the code. | ```sh $ ls $ echo $? # 0 $ fsdsgdg $ echo $? # 127 command not found $ exit 100 $ echo $? #100 ``` Special parameters: * `$?` - exit code of the last executed command * `$$` - process id of the shell * `$!` - process id of the job placed in background * `$0` - the name of the shell\ All params - ```sh $ echo $$ 56103 $ echo $0 -zsh ``` #### Tests Bash tests are constructs that allow you to do conditional expressions. Enclosed within square brackets with whitespace on either side (very important).\ The test utility evaluates the expression and, if it evaluates to *true*, returns a *zero (true)* exit status; otherwise it *returns 1 (false)*. ```sh $ [ 1 = 1 ] $ echo $? 0 $ [ 1 = 0 ] $ echo $? 1 # these are invalid uses of [ ] $ [1=1] # error because there are no whitespaces around [ and ] bash: [1=1]: command not found $ [ 1=3 ] # no error, but the result is always 0 0 ``` You can compare values with a variable too. ```sh $ a=20 $ [ $a == 20 ] $ echo $? 0 $ [ $a == 5 ] $ echo $? 1 ``` **Logical operators** `!` means `no`. `||` means `or`. `&&` means `and`. **\[\[ operator** It tolerates variables that don't exist and treat it as empty string. ```sh $ [ $nothing = 2 ] bash: [: =: unary operator expected $ [[ $nothing = 2 ]] $ echo $? 1 # exit code 1 (comparison is '' = 2) ``` **Unary operators** * `-z` returns true only if the argument is an empty string * `-a` returns true if the argument is a *file or directory* and it exists * `-d` returns true if the argument is a *directory* ```sh $ FOO=bar $ [ -z "$FOO" ] $ echo $? 1 $ unset FOO $ [ -z "$FOO" ] $ echo $? 0 $ ls dir file.txt $ [ -a file.txt ] # 0 $ [ -a dir ] #0 $ [ -a something.txt ] #1 $ [ -d dir ] #0 $ [ -d file.txt ] #1 ``` **if statements** `if` statements consist of a test (enclosed in `[[` and `]]`), followed by `then` and the commands to run if that conditions evaluates to `true`.\ There can be an optional `elif` that is executed if the condition evaluates to `false`. You end the if block with `fi` ```sh $ if [[ 10 -gt 2 ]] > then > echo "10 is greater than 2" > elif [[ 10 -lt 2 ]] > then > echo "10 is less than 2" > else > echo "They are equal" ``` `if` can be followed by any command. It will compare the exit code of the command to true/false ```sh $ if grep not_there /dev/null > then > echo found not_there > else > echo not_there is missing > fi not_there is missing ``` #### Loops **for loops** The `for` loops is like the regular c-style loop. You need to use `do` and `done` ```sh $ for (( i=0; i<3; i++ )) > do > echo $i > done $ 0 1 2 ``` You can use the `for in` construct too. ```sh $ ls 1.txt 2.txt 3.txt $ for f in $(ls *txt) > do > echo "File $f contains: $(cat $f)" > done File 1.txt contains: one File 2.txt contains: two File 3.txt contains: three ``` Bash has `while` and `until` too. Use `do` and `done` too. **case statements** ```sh $ cat script.sh #!/bin/bash echo "user input: $1" a=$1 case "$a" in 1) echo 'a is 1';; 2) echo 'a is 2';; *) echo 'a is not 1 or 2';; esac $ ./script.sh 23 user input: 23 a is not 1 or 2 $ ./script.sh 1 user input: 1 a is 1 $ ./script.sh 2 user input: 2 a is 2 ``` Case statements are often used to process command-line options. The helper builtin `getopts` is very useful here. #### set command `set` is a builtin that shows all the variables and functions set in your environment. You can run is `$ set`\ Note - `env` shows the *exported* variables only. #### File substitution We saw command substitution operator `$(....)` which substitutes the output of the process contained in it to the command\ File substitution operator `<()` does the same for file, i.e. it substitutes a file containing the output of the process contained within it ```sh $ mkdir a b $ echo a1 > a/1.txt $ echo b2 > b/2.txt $ diff <(ls a) <(ls b) 1c1 < 1.txt --- > 2.txt ``` #### Subshells It is a shell that inherits variables from the parent shell. It starts when you open a parentheses `(` and the running is delayed until the parentheses is closed with `)` ```sh $ OUTER=10 $ ( > echo "OUTER value: ${OUTER}" > echo "in the subshell" > INNER=20 > ) ABC value: 10 in the subshell $ echo "INNER value: ${INNER}" INNER value: # value is empty since the variable from the subshell is not available here $ echo "OUTER value: ${OUTER}" 10 ``` When you change folder when inside a subshell, then that folder change applies only within the subshell. This will help you avoid multiple `cd` `cd -` when you are in scripting mode. ```sh $ ls abc xyz $ pwd /Users/ann/dev/learn-bash $ ( > pwd > cd abc/ > pwd > ) /Users/ann/dev/learn-bash /Users/ann/dev/learn-bash/abc $ pwd /Users/ann/dev/learn-bash ``` #### Internal field separator (IFS) This is a variable that specifies the separator character for file names. By default it is (whitespace), and . ```sh But it can be modified. This is particularly helpful to prevent the common bug when filenames have spaces in them. $ touch "file with spaces.txt" $ set | grep IFS IFS=$' \t\n' $ for f in $(ls) > do > echo $f > done file with spaces.txt # change the internal file seprator - remove the space character IFS=$'\t\n' $ for f in $(ls) > do > echo $f > done file with spaces.txt ``` # git ### Good Articles * Simple list of git commands - * Git Rebase - ### How to Use Pull requests search filters Source: * To view the PRs that were closed with out merging `is:pr is:unmerged state:closed` * To see PRs that were closed without merging after a specific date `is:pr is:unmerged state:closed closed:>2018-04-01` ## How to Set the Correct Email to GitHub account When you switch between the public github and Intuit github, you may have trouble seeing the correct user associated with the commits. In order to avoid this mismatch, you need to set the correct git user BEFORE you make the commits. **Command to see the currently configure user** ```sh git config --global user.email ``` Replace `user.email` with your actual email address like so: `git config --global user.email "your_email@example.com"`. **Command to see all local configuration (can be run only from a git repo)** ```sh git config --local --list ``` ### The Best Solution Configure the local repos pointing to public GitHub to use Gmail address. Configure global value to be corporate email address. (Or the other way around) ```sh cd ~/dev/GitHub/my-learnings git config --local user.email git config --global user.email ``` #### Result: ```sh git config --local user.email ==> your-pers-email git config --global user.email ==> your-corp-email ``` # AI / ML / Gen AI A collection of good reference articles about Generative AI This is the home for a lot of information on Machine Learning, Generative AI and AI in general. Here is a curated set of good articles/videos on Gen AI and ML in general. There is a good mix of introductory material, hands-on and more in-depth resources. ### Good articles on Gen AI/ ML at introductory level * [A great intro of Gen AI (with good animation & examples)](https://ig.ft.com/generative-ai/) * [LLM explained with less jargon and math](https://www.understandingai.org/p/large-language-models-explained-with) * [A good summary of the important concepts in ML](https://ataiva.com/machine-learning-mastery-series/) * [\[1hr Talk\] Intro to Large Language Models](https://youtu.be/zjkBMFhNj_g?feature=shared) (Andrej Karpathy) ### Prompt Engineering Guides * [Open AI’s guide to Prompt engineering](https://platform.openai.com/docs/guides/prompt-engineering) * [A Beginner’s Guide To Prompt Engineering | ml-articles – Weights & Biases](https://wandb.ai/mostafaibrahim17/ml-articles/reports/A-Beginner-s-Guide-To-Prompt-Engineering--Vmlldzo2MzE1NTg2) * [Prompt engineering guide](https://www.promptingguide.ai/) * [Language Modeling: A Beginner's Guide | Language-Models – Weights & Biases](https://wandb.ai/madhana/Language-Models/reports/Language-Modeling-A-Beginner-s-Guide---VmlldzozMzk3NjI3) ### Build your own chatbot Open AI’s GPTs - custom personalized GPT - it is very easy to build - with a simple UI tool and can hook it up to any custom API - * [Introducing GPTs (openai.com)](https://openai.com/blog/introducing-gpts) * [ChatGPT (openai.com)](https://chat.openai.com/gpts/editor) * [A step-by-step guide to building a chatbot based on your own documents with GPT](https://bootcamp.uxdesign.cc/a-step-by-step-guide-to-building-a-chatbot-based-on-your-own-documents-with-gpt-2d550534eea5) ### Run other LLMs (free, open source) models on your local machine * [Llama 2 - Every Resource you need (philschmid.de)](https://www.philschmid.de/llama-2) * [Run Code Llama 70B on M2 Max machine](https://annjose.com/post/run-code-llama-70B-locally/) ### How to implement a transformer architecture * [Mat’s Blog - Transformers From Scratch (matdmiller.com)](https://blog.matdmiller.com/posts/2023-06-10_transformers/notebook.html) * [Let's build GPT: from scratch, in code, spelled out.](https://www.youtube.com/watch?v=kCc8FmEb1nY) (Andrej Karpathy) ### More detailed videos * [State of GPT | BRK216HFS](https://youtu.be/bZQun8Y4L2A?feature=shared) * [The spelled-out intro to language modeling: building makemore](https://www.youtube.com/watch?v=PaCmpygFfXo) ### Deploy ML models to Prod * [Production best practices to deploy ML models - OpenAI Guide](https://platform.openai.com/docs/guides/production-best-practices/mlops-strategy) * [Explaining the Microsoft 365 Copilot System - YouTube](https://www.youtube.com/watch?v=E5g20qmeKpg) - 2 min video of the architecture # Generative AI - A Primer Notes from reading a good introductory article to Generative AI ### Details about the article Title: **Generative AI exists because of the transformer**\ *Link:* [https://ig.ft.com/generative-ai/ ]()\ Author(s): [Visual Storytelling Team](https://www.ft.com/visual-and-data-journalism) and [Madhumita Murgia](https://www.ft.com/madhumita-murgia)\ Published: **9/11/2023** London \ Referred from: [HN post - How Transformers Work](https://news.ycombinator.com/item?id=37774676). ### Summary Generative AI - software that can create plausible and sophisticated text, images, code at a level that mimics human ability. LLMs are pattern-spotting engines that guess the next best option in a sequence. They are not search engines that are looking up facts. Process * *tokenizes* the text (breaks it down into sub-words) * observes that word in the *context of other occurrences* of the word * processes this data and produces a vector - **word embedding** - a list of numbers, based on each word's proximity to the word LLM is underpinned by the Transformer architecture proposed by Google researchers in 2017. Transformer architecture * processes a sequence of words together at a once, instead of each individual words separately. This helped LLMs to have better context and patterns, and thus producing *more accurate* text. And this process *runs faster* because it can be parallelized. * A key concept of the transformer architecture is **self-attention** which allows LLMs to understand *relationship* between words. It looks at *each token* in a body of text and and decided which of those words are most important to *understand its meaning*. * Prior to transformers, the standard method for language translation was **RNN (Recurrent Neural Network)**, which scanned each word sequentially - in the forward direction only. * With self-attention, transformers compute all the words at the same time. When the *user gives a prompt*, it tokenizes, encodes the prompt and represents it in a machine-understandable format - it includes the meaning, positions and relationship between words. Then it will try to predict the next word, the next word etc. until the output is complete. These predictions also come as tokens and the model assigns a *probability score* to each of these tokens - to indicate the likelihood that the token is the best next word. So the content generated by LLMs may seem plausible and coherent, but they *not always factually correct*. Transformer models can *recognize any repeating patterns* - pixels in image, code, notes in music, DNA in proteins etc. Two ways to predict these tokens: 1. *greedy search* - the model predicts each word in *isolation*. But this could make the whole phrase irrelevant, even when each individual token is meaningful. 2. *beam search* - the model looks at the probability of *a larger set of tokens* instead of each token individually. So it considers multiple routes and finds the best option. #### Further Reading * [Transformer: A Novel Neural Network Architecture for Language Understanding](https://blog.research.google/2017/08/transformer-novel-neural-network.html) ### Detailed Notes * LLM is a giant leap forward in our quest to build intelligence * Generative AI - software that can create plausible and sophisticated text, images, code at a level that mimics human ability. * LLM is underpinned by the Transformer architecture proposed by Google researchers in 2017 **How does LLM generate text?** First, it translates words into a format that it can understand. * a set of words are broken into tokens - **tokenization**. Tokens are usually sub-words. Example - `We go to work by train`. One of the tokens is `work`. * in order to understand the meaning of `work`, the LLM observes that word in the context of other occurrences of the word - using enormous sets of training data created from the internet. * after the training, there will be... * a few words that appear next to `work` - `are`, `her`, `friend`, `admirable`, `streamlined` * and other words that don't - `dove`, `polka` * the LLM processes this data and produces a vector - **word embedding** - a list of numbers, based on each word's proximity to the word `work`. * word embedding of `work` could be `[.35, .21, .07, .25, .33,....]` * each of these embeddings contain 100's of values, each of which represents various linguistic features of the word. * we don't know exactly what each of these values represent, but the words with similar embeddings are usually used in similar (comparable) context. * Example - `football` and `soccer` are not identical, but have similar meaning, so their embeddings quantify that closeness * if we take just two of these characteristics and project them to a 2-d plain, we can see the distance between them and can identify clusters of similar words. | | | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ![](https://674224673-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LK2e1elhchncfzxWFRk%2Fuploads%2FDVTk4pwxi3Txr3ZQgDKO%2FPasted%20image%2020240131165845.png?alt=media\&token=c2f4bf40-e13c-481a-9f75-c7aab44cc498) | ![](https://674224673-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LK2e1elhchncfzxWFRk%2Fuploads%2FXwxrbAd4wCe4ME2l1gdD%2FPasted%20image%2020240201111246.png?alt=media\&token=70501c1e-a846-4485-8b03-0ac9fd7eb5e6) | #### The key differentiator of LLMs - Transformer Architecture Transformer architecture processes a sequence of words together at a once, instead of each individual words separately. This helped LLMs to have better context and patterns, and thus producing *more accurate* text. And this process *runs faster* because it can be parallelized. This architecture was first published by Google Research team in 2017 - [Transformer: A Novel Neural Network Architecture for Language Understanding](https://blog.research.google/2017/08/transformer-novel-neural-network.html) A key concept of the transformer architecture is **self-attention** which allows LLMs to understand *relationship* between words. It looks at *each token* in a body of text and and decided which of those words are most important to *understand its meaning*. Prior to transformers, the standard method for language translation was **RNN (Recurrent Neural Network)**, which scanned each word sequentially - in the forward direction only. With self-attention, transformers compute all the words at the same time. Example - take the word `interest`. In the sentence `I have no interest in politics`, the word `interest` is used as a noun to indicate the subject's affiliation to politics. In the sentence `The bank's interest rates continue to rise`, the same word is used in the financial sense. Even when we combine the two usages, `I have no interest in hearing about the rising interest rate of the bank`, the model is able to recognize the meaning of the word in each context. In the first use of the word `intrest`, `no` and `in` gets the highest attention. For the second usage, it is `rate` and `bank`. It also allows the model to use other words in place of `interest` - at the right place. For example, `I have no enthusiasm in hearing about the rising...`. This is particularly useful when summarizing content. **Another example** - 1. `The dog chewed the bone because it was hungry`. Here, `it` refers to the dog. 2. `The dog chewed the bone because it was delicious`. Here, `it` refers to the bone, not the dog. This self-attention helps LLMs to gather context from a broad area - well beyond the sentence boundaries. This helps you scale things up. #### LLM's available now 1. **OpenAI**'s **GPT-4** 2. **Google** **PaLM** which powers its **Bard** chatbot (and now a newer model Gemini) 3. Anthropic Claude 4. Meta's LLaMA 5. Cohere's Command 6. Mistral #### How it predicts the next token LLMs are trained on huge corpus of text available in the internet. They identify patterns and context in this data and create word embeddings, positional encoding and self-attention. When the user gives a prompt, it tokenizes, encodes the prompt and represents it in a machine-understandable format - it includes the meaning, positions and relationship between words. Then it will try to predict the next word, the next word etc. until the output is complete. These predictions also come as tokens and the model assigns a probability score to each of these tokens - to indicate the likelihood that the token is the best next word. There are two ways to predict these tokens: 1. *greedy search* - the model predicts each word in isolation. But this could make the whole phrase irrelevant, even when each individual token is meaningful. 2. *beam search* - the model looks at the probability of a larger set of tokens instead of each token individually. So it considers multiple routes and finds the best option. Beam search produces better accurate results and more human-like text.

Beam search that looks at larger set of tokens

#### Limitations of LLMs LLMs are not search engines that look up facts. They are *pattern-spotting engines that guess the next best option in a sequence*. The output of the LLMs may seem plausible and coherent, but they may not be factually correct. So they can fabricate information in a process called **hallucination**. So they can make up references to articles that don't exist, wrong authors for papers etc. Companies are trying to limit the extend of this hallucination in a few ways: 1. put humans in the loop to give feedback and fill in the gaps in information - **RLHF (Reinforcement Learning with Human Feedback)** 2. a method called **grounding** - cross-checks the LLM's output against web search results and give citations so that people can verify. #### The potential of LLMs The power of LLMs go far beyond text. Transformer models can recognize any repeating patterns (pixels in image, code, notes in music, DNA in proteins). For decades, AI research had produced specialized models to summarize, translate, search and retrieve. Transformers unified them all into a single structure capable of doing multiple tasks. # Course Notes - Google ML Intro Course taken in Nov 2018. ### Course Details Title: **Introduction to Machine Learning** Link: ### Introduction Peter Norvig, Director of Research at Google gives the intro explaining how ML helps software engineers - reduce the time programming, customize products for specific people, complete seemingly impossible tasks. Also, philosophical reason - it changes the way you think about a problem. Focus shifts from mathematical science to a natural science, running experiments, use statistics instead of logic to analyze the results. ### Tools Collaboratory - Free Jupiter notebook environment that runs fully in the cloud. Pandas - Python library for data structures and data analysis. It uses **DataFrames** and **Series**. It can load an entire file into a DataFrame, show interesting statistics about data, plot a histogram etc., More info at . You can run code in this environment and see the results in real-time. ### Framing **Supervised ML** - System that combines inputs to provide useful predictions on data that was never seen before. * **Label** - target that we are predicting; eg: "spam", "not spam". It is the **y** variable in linear regression. * **Feature** - input variable that describes the data (eg: words in the email, to/cc fields, email headers). It is the **x** variable in simple linear regression. * **Example** - A particular instance of data. It can be **labeled** (includes the feature/s and the label) or **unlabeled** (contains feature, but no label). We train the models using labeled examples and then use that model to predict the label of unlabeled examples. * **Model** - defines the relationship between features and labels. eg: spam detection model may identify some features as strongly associated with spam. Models have two phased - Training (creating/learning the model) and Inference (apply the trained model to unlabeled examples). * Types of models * Regression model: predicts continuous value (price of house in CA) * Classification model: predicts discrete values (is the email spam? is this an image of dog/cat/lion) ### Descending into ML * Linear Regression - method of finding a straight line or hyperplane that fits a set of points in the best way. **y = wx + b** (w is the weight and b is the bias) * Loss - penalty of bad prediction. Number that indicates how bad the prediction was on a single example. * Goal of training a model is to find a set of weights and biases that have *low* loss *on average* across all examples. * One of the methods to represent Loss is: * **L2 Loss (**aka **Squared Loss):** the square of the difference b/w predicted value and observed value of label. MSE = Mean Squared Error; D = data set that contains N pairs of examples (x, y) $$ MSE = \frac{1}{N}\sum\_{(x,y) \epsilon D} (y - prediction(x))^2 $$ # How To Guides \#git, #nextjs, #react ## How to add an environment variable in a React/NextJS app * create a file named `.env` in the root folder of your web app (not inside `src` or `app`, instead in the top level folder) * add the variable in the file * in a React app, the name of the environment variable should start with `REACT` , eg: `REACT_MY_KEY=my-value` * in a NextJS app, it should start with `NEXT_PUBLIC_`, eg: `NEXT_PUBLIC_MY_KEY=my-value` * restart the dev server. This is very important - otherwise the new env variable won't be available to the runtime ## How to add a local git repo to GitHub * go into your folder and say `git init` * add a `.gitignore` file if you don't have one. * add the files to the local repo `git add .` * commit the first set `git commit -m "initial commit` * name the main branch as "main" `git branch -M main` * Go to and create a new repo. * to avoid mistakes, do NOT initialize the repo with README, gitignore and license files. You can add these later. * Add a remote pointing to the new github repo - `git remote add origin https://github.com/annjose/movie-mosaic.git` * Push the local to remote - `git push -u origin main` Reference - ## How to rewrite git commit history Situation: You have a git repo (public/private) and you want to change the contents or the commit message of a specific commit. Example: On 5 Jan 2024, you accidentally checked-in the API token for an API (TMDB API) t your public github repo annjose/learn-react and you want to remove that token (i.e delete the line from the file in that commit) Solution * read this reference [Git - Rewriting History (git-scm.com)](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History) * I also asked ChatGPT to give a solution and it gave a concise answer. Pairing that with the git documentation really helped. * one thing that was not clear in the article above is that **when** to run the command `git commit --amend` - before making the changes in the file or after. The answer is after. * also, the article seem to indicate that in the command `git rebase -i `, you should give the SHA of the commit that you want to modify. But in reality, if you do that, it shows the commits AFTER that commit. So you should either give the SHA of the commit that appears *before* the rogue commit (i.e. the rogue commit's parent) or give `HEAD~2`, `HEAD~3` etc. * Basically here is how you do it * do `git log` and find the rogue commit you want to modify * note the SHA of the *parent* commit of this rogue commit, eg: `f342b1` * run the command (in terminal or VS Code terminal) - `git rebase -i f342b1` * the terminal will open a vim-like editor and show the commits from the parent commit down to the latest commit in the repo * locate the rogue commit in this list and change the first word in that line from `pick` to `edit` (you may have to type `i` to switch to INSERT mode) * run `Esc` to exit the INSERT mode and then do `wq` to save the file and exit. If you want to exit without saving, you should do `q!` * At any point, if you want to abort this rebase operation, you can do `git rebase --abort` * make the changes in the files (remember - git is in a state that it has time-traveled to that rogue commit) * after making the changes (modify file, or rename file etc.), stage the changes (VS Code Source Control panel or the command `git add .`) * Run your application (app/NodeJS project) and confirm that the new changes are working fine. * Now you can commit the new changes by running `git commit --amend` (all the staged changes will be added to the rogue commit now) * Now continue with the rebase `git rebase --continue` to change any other commits that were marked with `edit` * If there are no more commits marked with `edit`, the terminal will appear normal. * when you do `git status` now, it will show that there are x number of changes on local main and the same x number of changes on remote main. * So we should push the local changes to remote *forcefully* by running `git push origin main --force` * Now the repo is in good shape. Go to guthub page and confirm that the rogue commit is fixed now ## How to amend the last commit Situation: You want to amend the last commit you made in a repo Solution * article reference * if you want to amend *only the commit message*, do `git commit --amend` The terminal will show a vim-like editor with the commit message at the top. You can update the message (use `i` to go into INSERT mode) * If you want to amend the *content of the commit*, then *before doing git commit --amend*, make the changes in the local files, then stage them (`git add .`) and then do `git commit --amend`. It will bring up the vim-like editor and show the commit message at the top. You can leave it as is or edit it. Save and Exit (`wq`) * If you want to amend *only the content and not the commit message*, you can run the command `git commit --amend --no-edit`- it will commit the staged changes without bringing up the vim editor to update the commit message. * Push the changes to remote (origin) forcefully `git push origin main --force`. (in GitHub desktop app, use the option 'Force push origin' at the top) # Good Reference Articles A curated list of good reference articles #graphql, #algorithms, #productivity ### **GraphQL** * Comparison of REST and GraphQL - ### Productivity * Time management for software engineers * * Use Pomodoro - 25 min work, 5 min break, 5 min context switching * 8 hrs = 13 poms; 1 pom for lunch; 5 poms for helping others. So 6 poms for work * set micro goals for each pom - keep writing what you are doing in the current pom * Master the interruptions - externalize your current context by taking good notes. This makes it easy to stop a pom at 25 min sharp and deal with interruptions (only critical ones) ### Algorithms & Data Structures * Swift Data Structures / Algorithms Book Safari Books - * Swift Algorithms Book Wayne Bishop - ### Miscellaneous * Martin Fowler's article on FeatureFlags - * Flutter (StopWatch App) - * Gmail addresses dots don't matter * Tech Interview Handbook - # Mobile Development # iOS iOS General concepts. Specific topics are described as sub-pages to this page. ## ARC Retain Cycles Source: * Strong reference - a reference that increments the reference count by 1 and thereby prevents the referring object to be deallocated by ARC. Strong references are safe to use when the object hierarchy is linear, i.e parent -> child only * Weak reference - reference that does not increment the ref count and thus does NOT protect the object from being deallocated by ARC. Good for child to refer to parent object - so when parent object goes out of scope, the child will also go out of scope automatically. In Swift, weak references have to be Mutable and Optional (because it should be possible to be set to nil after initial value). Use this for object-> delegate relationship and for objects that use NSNotificationCenter, NSTimer etc. * Retain cycle - Two objects have strong reference to each other. Both objects won't be deallocated. * Unowned reference - a reference that does not increment ref count (like weak reference), but is not an Optional (unlike weak reference). Use this only when you are sure that the object that defines the reference (for eg: closure) and the object that it refers to (instance that it captures) will be deallocated at the same time. For example, lazily defined closure properties. # Useful Information #### How to check if a View Controller is Presented Modally or Pushed to Nav Stack If you are presenting one VC directly on top of another VC, the property `isPresenting` will return true. If the VC is pushed to navigation stack, then the property will be false. However, if the second VC is wrapped inside a UINavigationController before it was presented, then the property will return false and hence cannot be used. The better alternative is to check if the first view controller in the navigation stack is `self` - if so, it means that the second VC was presented modally through an intermediate navigation controller. ```swift // If this view controller was presented modally (i.e. the first VC in its navigation // stack is itself), then show the Cancel button var isPresentedModally { if ((self.navigationController == nil) || (self.navigationController.viewControllers.first == self)) { return true } return false } ``` #### How to hide the "Back" button or the default back button text in the nav bar If a view controller's title is ver long, when more view controllers are pushed to the this VC's navigation stack, the navigation bar of those view controllers will display the text "Back" instead of this view controller's title. In most cases, this results in inconsistent UI in the app. So we can set the current view controller's back button title to be empty (so that the nav bar will show just the green Left arrow). ```swift // Inside MyViewController.viewDidLoad(:)..... backNavButton = UIBarButtonItem(title: "", style: .plain, target: nil, action: nil) self.navigationItem.backBarButtonItem = backNavButton ``` # Debugging Tricks ### See the Swift generated interface for Objective-C header file If you are accessing an Objective-C class from Swift, you can see the Swift interface of the ObjC file:\ \- Open the header file in Xcode, eg: MyClass.h\ \- Press the square button on top left of the main editor ("Related Items") and choose **Counterparts**\ \- You will see the item "MyClass.h (Swift 4.2 Interface)". This file will show the Swift interface ### Send deep link to an app in a simulator The target app's Info.plist should have the URL scheme `acj` registered. There should be a simulator running with the app installed. ``` xcrun simctl openurl booted "acj://message/1000" ``` ### See Background Color in UI views * Set a background color for all your UI views in the storyboard * Add all those views to a an IBOutlet Referencing Outlet collections (right click and choose "New Ref Out Coll..." * Then in the viewDidLoad of your view controller, iterate through the colors and set the background color to clear ```swift override func viewDidLoad\(\) { super.viewDidLoad\(\) // Clear background colors from labels and buttons. // backgroundColoredViews is a Reference Outlet collection created from the Storyboard for view in backgroundColoredViews { view.backgroundColor = UIColor.clear } } ``` # ViewController Concepts Notes from reading Apple's View Controller Programming guide. ## Role of ViewControllers Source: Apple's [View Controller Programming guide](https://developer.apple.com/library/archive/featuredarticles/ViewControllerPGforiPhoneOS/). * View Controller manages a portion of the application's user interface and its interaction with the underlying data. They also coordinate the transition between other parts of your UI. * Every view controller has a **single root view** to which you can add more views. * **UIViewController** class manages the views, handles events, transitions to other view controllers, coordinates with other parts of the app. * 2 type of view controllers: * **Content view controllers** - manages a specific part of the app's content * **Container view controllers** - has child view controllers and present them or facilitate navigation between them. Manages its own views (root view and sub views) as well as the *root views* of its child view controllers. It does not manage the content of its children; just the size and position of the *root view of those children*. Example: **UINavigationController, UISplitViewController** * There should be a clean separation between view controllers and data objects: * Data objects - contain logic to ensure integrity of data and manage the data * View controllers - validate inputs coming from views, package it in a way that the data objects understand and the reverse. Use *UIDocument* object to read and write data to persistent store. * Event Handling - View controllers are responder objects that handle events using delegate methods or action methods on the views * View controller owns all the views and any objects it creates. View-related objects will be automatically released by UIKit; but any objects explicitly created by View Controller should be managed by the view controller itself. * View controllers should present the views and adapt the presentation to match the underlying environment. * **Coarse-grained changes** and **fine-grained changes**: * coarse-grained changes happen when view controller's traits change. Traits are attributes that describe the overall environment like display scale. Two important traits - view controller's horizontal size class and vertical size class * fine-grained changes happen within given size class, for example when user rotates the device, the size class remains the same, but the screen dimensions change. Most of the time, using AutoLayout is sufficient to ensure that the views adjust themselves. ## View Controller Hierarchy * You should maintain the relationship between the view controllers as prescribed by UIKit to ensure that the automatic behaviors are delivered to the correct view controllers properly at runtime. * **Root View Controller** - This is the anchor of the view controller hierarchy. It defines the initial content seen by the user. Every window has exactly one root view controller. Window does not have any content of its own; the root view controller's content fills the window. Available as the property **rootViewController** of the **UIWindow** object. * **Container view controllers** - allows you to assemble complex UI from other view controllers. For eg: UINavigationController displays the content from a child view controller along with a navigation bar and toolbar. Container VCs are often installed as root VCs, but can also be presented modally or a children of other container VCs. * Some container controllers will not remove their child view controllers when it presents other view controllers. For example, navigation controller does not remove its child view controllers when a new VC is pushed to the stack. * You can implement a custom container view controller by explicitly adding and removing child view controllers. See details in * **Presented view controllers** - Presenting a view controller replaces the contents of the current view controller with the contents of the new VC, usually hiding the contents of current VC. * Usually used to present a UI as a modal; also used to build the application UI * when you present a VC, UIKit creates relationships between presented view controller and presenting view controller. See these properties in UIViewController.h * **presentedViewController** - the view controller that this view controller presented * **presentingViewController** - the view controller that presented this view controller * When you present a VC, UIKit looks for a view controller that can provide the context for presentation. Usually this is provided by a nearest container VC or the root VC or you can tell which view controller will handle the presentation context. * When a child VC of a navigation control asks UIKit to present a VC in full screen presentation style, the navigation controller provides the presentation context (instead of the child VS who initiated the request) so that the child VC doesn't have to know the bounds of the container. ## Guidelines and Best Practices 1. Use system supplied View Controllers - makes it easy to use and provides consistent experience to the user. eg: standard container view controllers like UINavigationController, UISplitViewController etc., UIImagePickerController in UIKit, contact info view controllers in Address Book UI f/w. 2. Do not modify the view hierarchy of system-provided view controllers. 3. Make each view controller an island - View controllers should not know about the internal workings or view hierarchy of other view controllers. If two VCs want to communicate with each other, do it using explicitly defined public headers, preferably using delegation design pattern. 4. Use the root view of your View Controller only as a container for other views. This will ensure that all views have a common root view and makes Auto Layout operations correct and simpler. 5. Know where your data lives - have a clean separation of responsibilities between view controller and data objects. VC's main responsibility is to ensure that its view contain correct info; it may save a cache of the data for performance or do validation of data; but it should never deal with managing the data or ensuring integrity of the data - that is the responsibility of the data objects. 6. View controllers should adapt to different sizes of screens and respond to runtime changes in size and size classes. # Nullability Annotations Two groups of nullability annotations -`_Nullable/_Nonnull` and `nullable/nonnull`. The lowercase one is more elegant and works for almost all scenarios: * Method declarations - write the annotation right after the open parenthesis of the parameter * `- (nullable NSString *) getItemWithIdentifier:(nonnull NSString*)identifier;` * Property declarations - write the annotation in the attributes list * `@property (copy, nonnull) NSString* name;` * You can also define Nullability regions to mark specific regions in the file to consider any pointer type to be nonnull. So you need to explicitly mark only `nullable` properties/parameters * `NS_ASSUME_NONNULL_BEGIN` and `NS_ASSUME_NONNULL_END` Source: # UI Testing XCTestUI #### Good Article References * **Waits in XCUI Test** (by Alex Ilyenko) - * **4 Tips to master XCTestExpectation** (Bla Bla Car Tech) - * **Asynchronous iOS testing using XCTWaiter** (by Shashikant Jagtap) - * **Reducing flakiness in Swift tests** by John Sundell - * **XCTest waiting** - # Xcode Concepts ## xcodebuild commands * `xcodebuild build`- builds the project to run on simulators (add scheme/workspace/project) * `xcodebuild analyze` - static analysis of the project * `xcodebuild test` - builds the project and runs the tests * `xcodebuild build-for-testing` - builds the project for testing (use with `test-without-building`) * `xcodebuild test-without-building` - run the tests on multiple simulators without building * `xcodebuild archive` - creates an xcarchive for the project * `xcodebuild -exportArchive` - exports a given archive as an IPA file Tool to upload the IPA to iTunes connect - ALTool (Application Loader Tool) Location of the tool: */Applications/Xcode.app/Contents/Applications/Application Loader.app/Contents/Frameworks/ITunesSoftwareService.framework/Support/altool* # Mobile Application Services This page compares the various application services that are available to mobile applications | Capability | Service | | | ------------------------------------ | ------------------------------------------------------------------------- | ------------ | | Feature flags / remote configuration | [Firebase Remote Config](https://firebase.google.com/docs/remote-config/) | LaunchDarkly | | AB Testing | [Firebase Remote Config](https://firebase.google.com/docs/remote-config/) | Optimizely | | | | | | Crash reporting | Firebase Crashalytics | | | End User Monitoring (EUM / APM) | ?? | AppDynamics | | Performance | Firebase Performance Monitoring? | | | | | | | | | | #### Firebase Test Firebase app to try all the capabilities - Firebase Remote Config * changes to your app's default behavior and appearance by changing service-side parameter values * provide variations on your app's user experience to different segments of your user base * random percentile targeting with Google Analytics for Firebase to A/B test improvements to your app across different segments of your user base # AB Testing Various options are available to do AB Testing - LaunchDarkly, Firebase Remote Config, Optimizely * LaunchDarkly * Firebase - * Optimizely - **Firebase Remote Config** Remote Config allows you to set defaults in your app that can be overridden from the Firebase console. With Analytics, you can segment your users and measure the impact of your A/B tests. Remote Config is Firebase's equivalent of LaunchDarkly. You can combine Remote Config and Google Analytics for Firebase to implement AB Testing in your app. Analytics allows you to define "user properties"; and then you can create conditions based on these user properties. You can combine these user properties with Remote Config rules (eg: OS type = iOS, country = Canada etc) Details - #### Good Articles * # Firebase Firebase provides a set of services for mobile applications. The services include Performance Tracking, TestLab, Storage, Authentication etc. One good thing is that each of these services are published as sub-specs so you can pick and choose the capabilities that you want.eg: `Firebase/Auth`, `Firebase/Core`, `Firebase/PerformanceMonitoringetc.` ### Remote Config Allows the configuration of the app to change based on users / segments etc. Equivalent to LaunchDarkly ### Performance Tracking * Provides automatic performance tracking for network calls * You can also track specific actions using `Performance.startTrace`, `Performance.incrementCounter`and `Performance.stop` #### Test Lab # Good Articles Blogs * UIStackView Tutorial by Ray Wenderlich - * Documentation markup in Swift - * xcodebuild command line to build/deploy iOS apps - * Xcodebuild command reference - * Test code that uses Singleton classes - * Define a protocol to abstract the operation that uses the Singleton class (eg: URLSession) * Define an extension to the Singleton class to implement the protocol * Modify the consuming class to inject the new protocol as a dependency with the default value set to the Singleton class * A clean way to localize strings using plist instead of .strings file - ## Good iOS Blogs to follow * Objc.io - [https://www.objc.io/issues/](https://www.objc.io/blog/) * Cocoa With Love - * NSHipster - * Ole Begemann - * Ray Wenderlich - * Swift by Sundell - * Mike Ash - * Krakendev (retain cycles, AutoLayout) - # iOS Troubleshooting ## Keyboard does not appear on Simulator when you tap inside a text field Root cause: One potential reason is that the hardware keyboard is enabled. You can confirm this by checking if NSNotification for keyboardWillShow and keyboardWillHide are mixed up (i.e when you tap inside a text field, the notification keyboardWillHide is sent instead of keyboardWillShow) Solution: Press Cmd+K (Toggle keyboard) on the simulator window. ## Unnecessary vertical blank space at the top of UITableView ## A table view looks crooked on iOS 10 only * Check if the tableView's estimatedRowHeight is set. On iOS 10, it has to be set explicitly sometimes. ## LaunchDarkly FeatureFlag Has Default Value, Not Actual Value * Make sure that you are pointing to the same LD environment as the QBSE server environment * Make sure that the Targeting is ON * Make sure that the user is not excluded by any rule * If you added a specific user-based rule, make sure that the userid is available in LD # Cocoapods Troubleshooting ### Cocoapods warning ALWAYS\_EMBED\_SWIFT\_STANDARD\_LIBRARIES When you run `pod install`, Cocoapods shows the following warning (usually this happens for projects that were recently moved to Cocoapods): ``` [!] The `URLSessionMockerSampleTests [Debug]` target overrides the `ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES` build setting defined in `Pods/Target Support Files/Pods-URLSessionMockerSampleTests/Pods-URLSessionMockerSampleTests.debug.xcconfig'. This can lead to problems with the CocoaPods installation ``` #### Root cause This warning indicates that the build setting `ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES` that Cocoapods wants to manage has already been set in the Xcode settings and hence may conflict with Cocoapod's setting. You can find this setting in Xcode (go to Build settings of the target(s) mentioned in the warning message (in this case `URLSessionMockerSampleTests [Debug]`); look for the setting `Always Embed Swift Standard Libraries` in the section `Build Options`). Usually, this warning appears when existing projects are converted to use Cocoapods, because this etting may already set in Xcode and now Cocoapods wants to manage it for all the frameworks that it brings into the app target. #### Solution There are two options (as the warning also indicates) - either delete the build setting from the target OR modify the build setting in Xcode to `$(inherited)` so that the value set by Cocoapods will take precedence. You can do this as follows: * In Xcode, go to the Target settings and find the build setting `Always Embed Swift Standard Libraries` * Either select the settings and hit the Delete key so that it is removed from the project file * OR click on the dropdown and select `Other..` so that a text popup appears. Set/replace the text with the value `$(inherited)`. Dismiss the text popup; make sure that the value is updated * Run `pod install` again. It should run without any warnings ### Unit Tests Fail Without Executing In some projects with specific Pods in Tests target, the tests will fail without even executing, i.e. Xcode test job shows "Test Failed" notification, but none of the tests would have executed. This happens even if there is only one empty test function in the whole project. The logs in Report navigator shows the following error: ``` URLSessionMockerSample.app (32192) encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying error: The test runner exited with code 1 before checking in.)) ``` I have seen this issue with a specific Cocoapod named `Mocker`. Here is the Podfile for that project: ``` target 'URLSessionMockerSample' do target 'URLSessionMockerSampleTests' do inherit! :search_paths # Pods for testing pod 'Mocker' end end ``` When I replaced `Mocker` pod with another Pod (eg: `AFNetworking`), the tests executed properly. There seems to something specific to that pod (although I couldn't find anything special in its pod spec). Two workarounds I found for this problem were: 1. Add the offending pod outside the Tests target - as follows ``` target 'URLSessionMockerSample' do pod 'Mocker' target 'URLSessionMockerSampleTests' do inherit! :search_paths # Pods for testing end end ``` 1. Change the structure of the Podfile such that the test targets are at the same level as the app target as follows: \`\`\` target 'URLSessionMockerSample' do end target 'URLSessionMockerSampleTests' do ## inherit! :search\_paths ## Pods for testing pod 'Mocker' end ```` ## Https Error while running `pod install` ```text [!] 'AppDynamicsAgent' uses the unencrypted http protocol to transfer the Pod. Please be sure you're in a safe network with only trusted hosts in there. Please reach out to the library author to notify them of this security issue. ```` The podspec of that pod would have a `source` attribute which points to an HTTP url instead of HTTPS, for example here is a snippet from the [podSpec of AppDynamics](https://github.com/CocoaPods/Specs/blob/d0ec5a65e80656c8d78e12ff19f251df879e0bc2/Specs/0/b/f/AppDynamicsAgent/42.9.0/AppDynamicsAgent.podspec.json) ``` "source": { "http": "http://cdn.appdynamics.com/eum-mobile/iOSAgent-4.2.9.0.zip" }, ``` # Miscellaneous ## Command Line - How To Change the Shell * Through System Preferences * Open Users & Groups, ctrl-click your username, * select “Advanced Options”. Select the Shell. * Through Command line * Run command chsh -s /usr/local/bin/ * Restart the Terminal ## CI/CD - TeamCity Pull Request Builds * TeamCity is working on making Pull Request as first class citizens - * Request to enable filtering of PRs based on target branch - * * Hadi Hariri's blog on automatically building PRs in TeamCity - [https://blog.jetbrains.com/teamcity/2013/02/automatically-building-pull-requests-from-github-with-teamcity/\\#comment-290](https://blog.jetbrains.com/teamcity/2013/02/automatically-building-pull-requests-from-github-with-teamcity//#comment-290) # Google Calendar API Google's calendar API can be used to get calendar events of a user after (s)he has authorized the app. ### Learnings * Very easy to use. The [Quickstart guide](https://developers.google.com/calendar/quickstart/ios) was very helpful * Tried it during the QBSE mobile hack-day 3/30/18 to match calendar events to unreviewed trips. * Made a working demo of the code * Pranay registered the iOS app in QBSE's Google developer account and thus got the client id. * here is the plist - [https://files.slack.com/files-pri/T7AN7J5RP-F9YMD0YDA/client\\\_1012179567841-13araloiq5ofplb08ovcmkup4vk4di70.apps.googleusercontent.com.plist](https://files.slack.com/files-pri/T7AN7J5RP-F9YMD0YDA/client/_1012179567841-13araloiq5ofplb08ovcmkup4vk4di70.apps.googleusercontent.com.plist) ### References * Quick start guide - * API explorer to try the APIs without writing any code - [https://developers.google.com/apis-explorer/\\#s/calendar/v3/calendar.events.list](https://developers.google.com/apis-explorer//#s/calendar/v3/calendar.events.list) * Google Places API - # Learning Resources \#web-dev ## Learning Path * [ ] Learn **JavaScript** * Course: *Learn JavaScript Online* (completed on xx Nov 2023) * Course: Execute Program *Modern JavaScript* (completed on xx Nov 2023) * [ ] Learn **TypeScript** * Course: Execute Program *TypeScript Basics* (completed on xx Nov 2023) * Course: Execute Program *Everyday TypeScript* * [ ] Learn **React** * Course: *Joy of React* by Josh * React course by React team (QuickStart, TicTac Toe) * React Foundations course by NextJS 4. Learn CSS ### Learning Resources #### JavaScript * Execute Program - Modern JavaScript * Execute Program - JavaScript Arrays * - use George's github account * - free * * Josh Comeau's courses - #### CSS * Tailwind - * Bootstrap - React * React course by React team (QuickStart, TicTac Toe) # Troubleshooting Issues Common issues you would find working with TypeScript, Next.js and how to solve them. ## TypeScript shows file not found error in tsconfig.json Situation: You move or rename a TypeScript file inside VS Code and immediately `tsconfig.json` shows an error 'File xyz.tsx not found.\` Rootcause: This happens because TypeScript server did not record the move/rename operation. Solution: This is a known issue being discussed in this PR and the correct solution is not available (as of Feb 2024). But there is a workaround - restart the TypeScript server. * In VS Code, open command pallette `Ctrl+Shift+p` and choose the command `Typescript: Restart TS Server`).