The end is near - LISA's final day is here. Looking forward to performance tuning labs and advice on training others.

Tracing Lab

https://www.usenix.org/conference/lisa18/presentation/sharma-1

Suchakrapani Sharma - Scientist at ShiftLeft

Geneviève Bastien - Research Associate at Dorsal Laboratory of École Polytechnique de Montréal

Mohamad Gebai - Software Engineer at SUSE

Overview

Low impact recording on high frequency events such as syscalls, network events, scheduling, interrupts. Useful for debugging performance issues

when to trace

Tracing can be useful for some scenarios, such as below:

When you want low overhead to gather maximum data
You want exact flow of events at a granular level
Intermittent bugs
Complex systems with many moving parts and interdependencies
All other approaches have failed

What to trace

Service level
Application level
Operating System level

Tutorials

Available at

https://github.com/tuxology/tracevizlab

Writing a good training course

Speaker: Branson Matheson (Cisco)

Effective Training

What does this mean:

Make it memorable
Get people involved
Useful skills

Components of creating a training program.

I have an idea
Creating exercises
Scoping the class
Make it awesome
Building the outline
Create a proposal
Create a Proposal

I have an idea

Take a basic idea. Doesn't need to be complex. Start to wordsmith it, brainstorm it. Make a word-cloud, think about all the things which might be interesting. Don't restrict yourself. This creates a good starting point.

Define an objective. Use the wordsmith, highlight what is important and cross out anything which is not important to this objective.

Scoping

Sometimes the class is created for a specific situation (e.g. a conference). Other times, it is a passion you "make it work" for something else.

Use your wordsmith and further narrow down the scope. Look for the linking topics. Start to think about a flow

Considerations

How familiar are you with the material? There are bound to be lots of questions
How hard is it to develop methods to teach the idea?
How long will the class be?
How many students will you support?
Who will be attending?
Will you have a co-instructor to help?
Do you have a venue in mind? Will your class fit in the venue?What formats will you use for exercises? (individual or team, cooperative vs competitive, open discussion vs partner reviewing each other's work)
Constrain the content to suit the audience

Outline

Look for linking topics around your main idea
Organise your thoughts into idead
Take time to flesh out each item
Don't be afraid to be wordy at this stage (you can take out the extra words later)
Reorganise the sections to build on each other, try to develop a story.
Create as broadly as you can (experience-wise). This will allow you to tailor each presentation for each venue.

The Presentation

Intro the class and yourself
Take each concept and supporting info and make sections
Plan sections around breaks (exercises are a great place to start or end with a break)
Leave time at the end for informal discussion (i.e. after the class ends).

Introduction

The class (maybe brief ice breaker)
Yourself (bit of a bio)
Time Schedule
Obligations (cellphones, questions, disclaimers, if no photography, if u need to leave the room etc)
Outline (what we are covering)

Sections

Say it, teach it, say it

That final section (recap) allows for questions if people didn't understand

Keep your concepts simple
Exercises should be after questions

Exercises

Only 5 - 10 minutes per exercise.
They should be able to complete exercises with the knowledge of the class
Should be self-contained, limit external dependencies (avoid relying on internet etc)
Exercises should be practically useful within their workplace.

Individual Exercises

Start with writing exercises
Make them relatable to their own environment
Less is more, ask for distinct things the students have to thik about
Have students read their answers and defend them

Group exercises

Cooperative or competitive
Encourage discussion within groups
Try to pair most experienced with least when creating groups

Exercise Environments

If you need an environment, consider a fixed configuration. For instance, Vagrant, an OVA etc. Try not to rely on internet (use an ISO?) - distribute a USB key or ensure you can download from LAN. This can be an issue for disk space and memory on laptops.
Consider a pre-established environment
Support wireless and wired connections

Test

Test, test and test
Get someone else to test

Make it Awesome

Each slide should be a simple, concise story (can digest in 10 - 15 seconds)
The details will come from the teacher. The summary should be on the slide.
Don't "read the slide"
Take key concepts and add colour to highlight, add motion to garner attention, avoid sound effects
Supporting graphics - either make it the background or the central concept.
Transitions should enhance, not distract. Big transitions for topic changes. Small transitions for slide changes within the same topic. Very short transitions

Writing the proposal

You need to sell it to students. Remember they are looking sometimes for content, for specific presenters, for style, etc
Write the student section first (what they are lookg for etc)
Trite a title relevent to the students (short and concise, active words etc)
Write a description (what are you teaching, why are you teaching this topic, have a good tag line at the end, something memorable)
Write your bio (relevant experience, who you are (not just what you are)
Check for spelling and grammar
Keep your tutorial up to date, don't just recycle it for many years.

Talk - How do you present a class

https://www.usenix.org/conference/lisa18/presentation/matheson-0

Speaker - Branson Matheson - Cisco

High level approach

Think
Prepare
Execute

When you give a class, it is a performance. You are the performer. You need to be like an entertainer. If you need sugar / caffeine etc to be bouncing off the walls, do it! You need to be positive.

A good course needs to be engaging and entertaining.

You need to keep students attention even when it is hard (tired, difficult content, etc)

A good class leaves a lasting impression

Performer

Commands the room
Engages individuals
Moves around the room
Keeps a positive view (even if you make a mistake, the audience might not realise)
Always face the audience
Emphasise important points
A good instructor can gauge the audience and evolve the class to meet the needs
Makes everyone feel welcome
Can work with a group but provide individual satisfaction (especially with questions/comments from audience)

Preparing for a class

Questions to answer before teaching

Audience
Standards
Timing (first class of the day, last one of the conference, etc)
Locations
Language
Expectations

Expect the unexpected

Demos don't work
Hardware issues
Questions you can't answer
Interruptions (fire alarms, power outages, etc)

Presentation Software

An instructor's view is critical. Notes will help you with current slide
Next slide helps
Having timing on the view helps too

Presentation Tools

Clickers are great but hackable
Bluetooth won't reach the entire classroom
Cellphone based controllers are clunky but useful
Smart Watches work well, but look awkward
Keep in mind security - laptops etc may be stolen from venues

Presentation video

Bring video converters / cable kit
Find out in advance about video projector
Turn off EVERYTHING else on your laptop (screen saver, popups, kill other apps, updates etc)

Presentation Audio

Podium mic or wireless
Jack for laptop
Are you being recorded? Are you OK with that?

Classroom Space

Podium? Feel free to move it for what works for you
Student tables, with power?
Is there room to move or are you constrained?
Pad / Easel - write down any points the students raise

Classroom Tools

Do you provide environments for the students?

Power (for that country)
Connectivity (wifi)
Consider shipping requirements (equipment, materials etc)

Execute

Stage

set the time wth walk-in, break music or video - break the ice, use energy
Consider placing workbooks at the front, they need to introduce themselves before you give them the book.
Introduce yourself.
Encourage Feedback

Engage

Try to spend a higher percentage of time directly working with the students than presenting.
Look for ways to hilight the concepts. e.g. at hacking talk - make a tinfoil hat; at social engineering talk - ask them to look you up - most interesting nugget).
Beef up your slides, make them simple but amazing
add audio / video
Write notes for your slides
Use multiple examples (you may need to discard some, certain examples may resonate better with other people)
Ask questions, involve people
Walk around the classroom
Humour is fantastic for emphasis, can be dangerous - avoid offence. If you tell a joke and nobody gets it, you're going to sweat.
Stand so that you're the focus. You're trying to make a personal relationship with people.
If someone is asking a question, focus on them completely.
If a student makes a wrong point - don't make them wrong. Ask them to defend it, understand how they came to that conclusion. Diversity of opinion is OK. Agree to disagree if you need to, or take it offline.
Always repeat the question (even if they use a microphone)
Thank the student for their question
Give the best answer that fits the whole class (sometimes you don't need to go deep, you can take the discussion offline)

reward

Reward good questions
Reward good answers
Reward valid comments/opinions
Reward class participation

Resolve

At the end of every session, ask for questions
Limit the scope and time for questions
Some questions are hard to answer "in the moment". It is OK to ask for time to research later. Make sure you share the info later. Also write the question down, this may come up later.
Ask for feedback. You want positive and negative feedback, ask for detailed comments
Encourage discussion after class (hallway track, email/twitter/etc)

Talk - Linux Troubleshooting

https://www.usenix.org/conference/lisa18/presentation/uphill

Speaker: Thomas Uphill - Narrabilis

Presentation can be found here:

https://goo.gl/qc8rHw

UNIX philosophy:

Do one thing and do it well.

Linux still adopting this - lots of commands and libraries which do one thing really well.

Linux briefly

Hardware - only kernel interfaces with hardware
Applications interface with kernel via system calls (via system libraries)

UNIX is like an onion

many layers
You can go too deep if you're not careful

Shared Libraries

Reason for this is to not reinvent the wheel.

Standard C library has the original 34 UNIX syscalls (malloc, socket, send, open, read, write, free etc). plus newer syscalls (about 200 in total these days)

LDD is the shared library linker (configured in /etc/ld.so.conf/)

/etc/ld.so.cache is generated containing all libraries it knows about in the active state. This can be queried faster than individually looking up the libraries.

LD_DEBUG=subcommand

For example (truncated output)

[root@localhost ~]# LD_DEBUG=help /bin/bash
Valid options for the LD_DEBUG environment variable are:

  libs        display library search paths
  reloc       display relocation processing
  files       display progress for input file
  symbols     display symbol table processing
  bindings    display information about symbol binding
  versions    display version dependencies
  scopes      display scope information
  all         all previous options combined
  statistics  display relocation statistics
  unused      determined unused DSOs
  help        display this help message and exit

To direct the debugging output into a file instead of standard output
a filename can be specified using the LD_DEBUG_OUTPUT environment variable.
[root@localhost ~]# LD_DEBUG=statistics /bin/bash
      2222:
      2222:     runtime linker statistics:
      2222:       total startup time in dynamic loader: 1684650 cycles
      2222:                 time needed for relocation: 1137576 cycles (67.5%)
      2222:                      number of relocations: 141
      2222:           number of relocations from cache: 4
      2222:             number of relative relocations: 2705
      2222:                time needed to load objects: 242198 cycles (14.3%)
      2223:
      2223:     runtime linker statistics:
      2223:       total startup time in dynamic loader: 740862 cycles
      2223:                 time needed for relocation: 275532 cycles (37.1%)
      2223:                      number of relocations: 141
      2223:           number of relocations from cache: 4
      2223:             number of relative relocations: 2705
      2223:                time needed to load objects: 208944 cycles (28.2%)
      2224:

ltrace

strace is good for syscalls

ltrace shows which library calls ahve been made

man pages for library calls

e.g.

# man gethostbyname

GETHOSTBYNAME(3)           Linux Programmer's Manual          GETHOSTBYNAME(3)

NAME
       gethostbyname,   gethostbyaddr,   sethostent,  gethostent,  endhostent,
       h_errno, herror, hstrerror, gethostbyaddr_r, gethostbyname2, gethostby‐
       name2_r, gethostbyname_r, gethostent_r - get network host entry

SYNOPSIS
       #include <netdb.h>
       extern int h_errno;

       struct hostent *gethostbyname(const char *name);

Linux boot process

BIOS --> Boot Sector --> Bootloader --> kernel --> kthreadd –> kernel daemons --> initrd --> mounts --> services --> etc

kernel state

Look at state dirs

devfs (/dev)
procfs (/proc)
sysfs (/sys)

each process shows in /proc/{pid}

/proc/cmdline shows kernel args

/proc/net shows a bunch of stuff

etc

[root@localhost fd]# pwd
/proc/976/fd
[root@localhost fd]# ll
total 0
lr-x------. 1 root root 64 Oct 31 15:10 0 -> /dev/null
lrwx------. 1 root root 64 Oct 31 15:10 1 -> socket:[16198]
l-wx------. 1 root root 64 Oct 31 15:10 10 -> pipe:[16768]
l-wx------. 1 root root 64 Oct 31 15:10 11 -> /dev/cpu_dma_latency
lrwx------. 1 root root 64 Oct 31 15:10 12 -> anon_inode:[eventpoll]
lr-x------. 1 root root 64 Oct 31 15:10 13 -> /dev/urandom
lrwx------. 1 root root 64 Oct 31 15:10 2 -> socket:[16198]
l-wx------. 1 root root 64 Oct 31 15:10 3 -> /var/log/tuned/tuned.log
lrwx------. 1 root root 64 Oct 31 15:10 4 -> socket:[16727]
lrwx------. 1 root root 64 Oct 31 15:10 5 -> anon_inode:[eventfd]
lrwx------. 1 root root 64 Oct 31 15:10 6 -> /tmp/ffi0BhT6y (deleted)
lrwx------. 1 root root 64 Oct 31 15:10 7 -> anon_inode:[eventfd]
lrwx------. 1 root root 64 Oct 31 15:10 8 -> socket:[16749]
lr-x------. 1 root root 64 Oct 31 15:10 9 -> pipe:[16768]
[root@localhost fd]# cat ../cmdline
/usr/bin/python-Es/usr/sbin/tuned-l-P[root@localhost fd]#

If you're wondering how many threads are running:

[root@localhost task]# pwd
/proc/976/task
[root@localhost task]# ll
total 0
dr-xr-xr-x. 6 root root 0 Oct 31 15:34 1178
dr-xr-xr-x. 6 root root 0 Oct 31 15:34 1183
dr-xr-xr-x. 6 root root 0 Oct 31 15:34 1187
dr-xr-xr-x. 6 root root 0 Oct 31 15:34 1191
dr-xr-xr-x. 6 root root 0 Oct 31 15:34 976

ELF

Ececutable and Linkable Format

PID relationships

If child is running but parent has died, it is orphaned

Parents wait for their children
Children die, parents wait for their status
Until the parent receives the status, the child is not truly dead...zombie (defunct)

a zombie process or defunct process is a process that has completed execution but still has an entry in the process table: it is a process in the "Terminated state". Parent processes always wait for their children. If the child process dies, the parent waits for the status. Until this is received, the child is not dead, it is zombie/defunct. The zombied process is not running in user space, it is in the kernel space.

A zombie process can only die if the parent dies.

Double Fork

A process forks and forks again.

This is useful to ensure there's no parent process, preventing zombies. The resulting process becomes a grandchild, not a child.

inodes

stat

can be used to show statistics on a file

[root@localhost task]# ln /etc/resolv.conf /tmp/resolv.conf
[root@localhost task]# ln -s /etc/resolv.conf /tmp/resolv1.conf
[root@localhost task]# stat /etc/resolv.conf
  File: ‘/etc/resolv.conf’
  Size: 68              Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 136459736   Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:net_conf_t:s0
Access: 2018-10-31 15:10:23.341000000 -0400
Modify: 2018-10-31 15:10:23.335000000 -0400
Change: 2018-10-31 16:01:28.755488176 -0400
 Birth: -
[root@localhost task]# stat /tmp/resolv.conf
  File: ‘/tmp/resolv.conf’
  Size: 68              Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 136459736   Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:net_conf_t:s0
Access: 2018-10-31 15:10:23.341000000 -0400
Modify: 2018-10-31 15:10:23.335000000 -0400
Change: 2018-10-31 16:01:28.755488176 -0400
 Birth: -
[root@localhost task]# stat /tmp/resolv2.conf
stat: cannot stat ‘/tmp/resolv2.conf’: No such file or directory
[root@localhost task]# stat /tmp/resolv1.conf
  File: ‘/tmp/resolv1.conf’ -> ‘/etc/resolv.conf’
  Size: 16              Blocks: 0          IO Block: 4096   symbolic link
Device: fd00h/64768d    Inode: 1171458     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Context: unconfined_u:object_r:user_tmp_t:s0
Access: 2018-10-31 16:01:36.422488176 -0400
Modify: 2018-10-31 16:01:36.422488176 -0400
Change: 2018-10-31 16:01:36.424488176 -0400
 Birth: -

limits

Controlled by PAM

When a process is running, /proc/{pid}/limits shows the limits applicable to that PID

troubleshooting - sources of problems

kernel
Processes
Users/Groups
Permissions
Space / Inodes
Capacity (memory / CPU / network IO)
NSS
Network

Steps:

Verify the problem
Read Logs
Check permissions
Check space
Trace
Check Ports
Trust nobody! (check things again for yourself)

Hostname lookups

use getent - it looks up hostnames the way the system would

(unless you're specifically checking DNS - in which case use dig / nslookup)