Mohamad Gebai - Software Engineer at SUSE

Overview

Low impact recording on high frequency events such as syscalls, network events, scheduling, interrupts. Useful for debugging performance issues

when to trace

Tracing can be useful for some scenarios, such as below:

• When you want low overhead to gather maximum data

• You want exact flow of events at a granular level

• Intermittent bugs

• Complex systems with many moving parts and interdependencies

• All other approaches have failed

What to trace

• Service level

• Application level

• Operating System level

Tutorials

Available at

https://github.com/tuxology/tracevizlab

Writing a good training course

Speaker: Branson Matheson (Cisco)

Effective Training

What does this mean:

• Make it memorable

• Get people involved

• Useful skills

Components of creating a training program.

• I have an idea

• Creating exercises

• Scoping the class

• Make it awesome

• Building the outline

• Create a proposal

• Create a Proposal

I have an idea

Take a basic idea. Doesn't need to be complex. Start to wordsmith it, brainstorm it. Make a word-cloud, think about all the things which might be interesting. Don't restrict yourself. This creates a good starting point.

Define an objective. Use the wordsmith, highlight what is important and cross out anything which is not important to this objective.

Scoping

Sometimes the class is created for a specific situation (e.g. a conference). Other times, it is a passion you "make it work" for something else.

Use your wordsmith and further narrow down the scope. Look for the linking topics. Start to think about a flow

Considerations

• How familiar are you with the material? There are bound to be lots of questions

• How hard is it to develop methods to teach the idea?

• How long will the class be?

• How many students will you support?

• Who will be attending?

• Will you have a co-instructor to help?

• Do you have a venue in mind? Will your class fit in the venue?What formats will you use for exercises? (individual or team, cooperative vs competitive, open discussion vs partner reviewing each other's work)

• Constrain the content to suit the audience

Outline

• Look for linking topics around your main idea

• Organise your thoughts into idead

• Take time to flesh out each item

• Don't be afraid to be wordy at this stage (you can take out the extra words later)

• Reorganise the sections to build on each other, try to develop a story.

• Create as broadly as you can (experience-wise). This will allow you to tailor each presentation for each venue.  

The Presentation

• Intro the class and yourself

• Take each concept and supporting info and make sections

• Plan sections around breaks (exercises are a great place to start or end with a break)

• Leave time at the end for informal discussion (i.e. after the class ends).

Introduction

• The class (maybe brief ice breaker)

• Yourself (bit of a bio)

• Time Schedule

• Obligations (cellphones, questions, disclaimers, if no photography, if u need to leave the room etc)

• Outline (what we are covering)

Sections

• Say it, teach it, say it

That final section (recap) allows for questions if people didn't understand

• Keep your concepts simple

• Exercises should be after questions

Exercises

• Only 5 - 10 minutes per exercise.

• They should be able to complete exercises with the knowledge of the class

• Should be self-contained, limit external dependencies (avoid relying on internet etc)

• Exercises should be practically useful within their workplace.

Individual Exercises

• Start with writing exercises

• Make them relatable to their own environment

• Less is more, ask for distinct things the students have to thik about

• Have students read their answers and defend them

Group exercises

• Cooperative or competitive

• Encourage discussion within groups

• Try to pair most experienced with least when creating groups

Exercise Environments

• If you need an environment, consider a fixed configuration. For instance, Vagrant, an OVA etc. Try not to rely on internet (use an ISO?) - distribute a USB key or ensure you can download from LAN. This can be an issue for disk space and memory on laptops.

• Consider a pre-established environment

• Support wireless and wired connections

Test

• Test, test and test

• Get someone else to test

Make it Awesome

• Each slide should be a simple, concise story (can digest in 10 - 15 seconds)

• The details will come from the teacher. The summary should be on the slide.

• Don't "read the slide"

• Take key concepts and add colour to highlight, add motion to garner attention, avoid sound effects

• Supporting graphics - either make it the background or the central concept.

• Transitions should enhance, not distract. Big transitions for topic changes. Small transitions for slide changes within the same topic. Very short transitions

Writing the proposal

• You need to sell it to students. Remember they are looking sometimes for content, for specific presenters, for style, etc

• Write the student section first (what they are lookg for etc)

• Trite a title relevent to the students (short and concise, active words etc)

• Write a description (what are you teaching, why are you teaching this topic, have a good tag line at the end, something memorable)

• Write your bio (relevant experience, who you are (not just what you are)

• Check for spelling and grammar

• Keep your tutorial up to date, don't just recycle it for many years.

Talk - How do you present a class

https://www.usenix.org/conference/lisa18/presentation/matheson-0

Speaker - Branson Matheson - Cisco

High level approach

• Think

• Prepare

• Execute

When you give a class, it is a performance. You are the performer. You need to be like an entertainer. If you need sugar / caffeine etc to be bouncing off the walls, do it! You need to be positive.

A good course needs to be engaging and entertaining.

You need to keep students attention even when it is hard (tired, difficult content, etc)

A good class leaves a lasting impression

Performer

• Commands the room

• Engages individuals

• Moves around the room

• Keeps a positive view (even if you make a mistake, the audience might not realise)

• Always face the audience

• Emphasise important points

• A good instructor can gauge the audience and evolve the class to meet the needs

• Makes everyone feel welcome

• Can work with a group but provide individual satisfaction (especially with questions/comments from audience)

Preparing for a class

Questions to answer before teaching

• Audience

• Standards

• Timing (first class of the day, last one of the conference, etc)

• Locations

• Language

• Expectations

Expect the unexpected

• Demos don't work

• Hardware issues

• Questions you can't answer

• Interruptions (fire alarms, power outages, etc)

Presentation Software

• An instructor's view is critical. Notes will help you with current slide

• Next slide helps

• Having timing on the view helps too

Presentation Tools

• Clickers are great but hackable

• Bluetooth won't reach the entire classroom

• Cellphone based controllers are clunky but useful

• Smart Watches work well, but look awkward

• Keep in mind security - laptops etc may be stolen from venues

Presentation video

• Bring video converters / cable kit

• Find out in advance about video projector

• Turn off EVERYTHING else on your laptop (screen saver, popups, kill other apps, updates etc)

Presentation Audio

• Podium mic or wireless

• Jack for laptop

• Are you being recorded? Are you OK with that?

Classroom Space

• Podium? Feel free to move it for what works for you

• Student tables, with power?

• Is there room to move or are you constrained?

• Pad / Easel - write down any points the students raise

Classroom Tools

Do you provide environments for the students?

• Power (for that country)

• Connectivity (wifi)

• Consider shipping requirements (equipment, materials etc)

Execute

Stage

• set the time wth walk-in, break music or video - break the ice, use energy

• Consider placing workbooks at the front, they need to introduce themselves before you give them the book.

• Introduce yourself.

• Encourage Feedback

Engage

• Try to spend a higher percentage of time directly working with the students than presenting.

• Look for ways to hilight the concepts. e.g. at hacking talk - make a tinfoil hat; at social engineering talk - ask them to look you up - most interesting nugget).

• Beef up your slides, make them simple but amazing

• add audio / video

• Write notes for your slides

• Use multiple examples (you may need to discard some, certain examples may resonate better with other people)

• Ask questions, involve people

• Walk around the classroom

• Humour is fantastic for emphasis, can be dangerous - avoid offence. If you tell a joke and nobody gets it, you're going to sweat.

• Stand so that you're the focus. You're trying to make a personal relationship with people.

• If someone is asking a question, focus on them completely.

• If a student makes a wrong point - don't make them wrong. Ask them to defend it, understand how they came to that conclusion. Diversity of opinion is OK. Agree to disagree if you need to, or take it offline.

• Always repeat the question (even if they use a microphone)

• Thank the student for their question

• Give the best answer that fits the whole class (sometimes you don't need to go deep, you can take the discussion offline)

reward

• Reward good questions

• Reward good answers

• Reward valid comments/opinions

• Reward class participation

Resolve

• At the end of every session, ask for questions

• Limit the scope and time for questions

• Some questions are hard to answer "in the moment". It is OK to ask for time to research later. Make sure you share the info later. Also write the question down, this may come up later.

• Ask for feedback. You want positive and negative feedback, ask for detailed comments

• Encourage discussion after class (hallway track, email/twitter/etc)

Talk - Linux Troubleshooting

https://www.usenix.org/conference/lisa18/presentation/uphill

Speaker: Thomas Uphill - Narrabilis

Presentation can be found here:

https://goo.gl/qc8rHw

UNIX philosophy:

Do one thing and do it well.

Linux still adopting this - lots of commands and libraries which do one thing really well.

Linux briefly

• Hardware - only kernel interfaces with hardware

• Applications interface with kernel via system calls (via system libraries)

UNIX is like an onion

• many layers

• You can go too deep if you're not careful

Shared Libraries

Reason for this is to not reinvent the wheel.

Standard C library has the original 34 UNIX syscalls (malloc, socket, send, open, read, write, free etc). plus newer syscalls (about 200 in total these days)

LDD is the shared library linker (configured in /etc/ld.so.conf/)

/etc/ld.so.cache is generated containing all libraries it knows about in the active state. This can be queried faster than individually looking up the libraries.

LD_DEBUG=subcommand

For example (truncated output)

[root@localhost ~]# LD_DEBUG=help /bin/bash

Valid options for the LD_DEBUG environment variable are:

  libs        display library search paths

  reloc       display relocation processing

  files       display progress for input file

  symbols     display symbol table processing

  bindings    display information about symbol binding

  versions    display version dependencies

  scopes      display scope information

  all         all previous options combined

  statistics  display relocation statistics

  unused      determined unused DSOs

  help        display this help message and exit

To direct the debugging output into a file instead of standard output

a filename can be specified using the LD_DEBUG_OUTPUT environment variable.

[root@localhost ~]# LD_DEBUG=statistics /bin/bash

      2222:

      2222:     runtime linker statistics:

      2222:       total startup time in dynamic loader: 1684650 cycles

      2222:                 time needed for relocation: 1137576 cycles (67.5%)

      2222:                      number of relocations: 141

      2222:           number of relocations from cache: 4

      2222:             number of relative relocations: 2705

      2222:                time needed to load objects: 242198 cycles (14.3%)

      2223:

      2223:     runtime linker statistics:

      2223:       total startup time in dynamic loader: 740862 cycles

      2223:                 time needed for relocation: 275532 cycles (37.1%)

      2223:                      number of relocations: 141

      2223:           number of relocations from cache: 4

      2223:             number of relative relocations: 2705

      2223:                time needed to load objects: 208944 cycles (28.2%)

      2224:

ltrace

strace is good for syscalls

ltrace shows which library calls ahve been made

man pages for library calls

e.g.

# man gethostbyname

GETHOSTBYNAME(3)           Linux Programmer's Manual          GETHOSTBYNAME(3)

NAME

       gethostbyname,   gethostbyaddr,   sethostent,  gethostent,  endhostent,

       h_errno, herror, hstrerror, gethostbyaddr_r, gethostbyname2, gethostby‐

       name2_r, gethostbyname_r, gethostent_r - get network host entry

SYNOPSIS

       #include <netdb.h>

       extern int h_errno;

       struct hostent *gethostbyname(const char *name);

Linux boot process

BIOS --> Boot Sector --> Bootloader --> kernel --> kthreadd –> kernel daemons --> initrd --> mounts --> services -->      etc

kernel state

Look at state dirs

• devfs (/dev)

• procfs (/proc)

• sysfs (/sys)

each process shows in /proc/{pid}

/proc/cmdline shows kernel args

/proc/net shows a bunch of stuff

etc

[root@localhost fd]# pwd

/proc/976/fd

[root@localhost fd]# ll

total 0

lr-x------. 1 root root 64 Oct 31 15:10 0 -> /dev/null

lrwx------. 1 root root 64 Oct 31 15:10 1 -> socket:[16198]

l-wx------. 1 root root 64 Oct 31 15:10 10 -> pipe:[16768]

l-wx------. 1 root root 64 Oct 31 15:10 11 -> /dev/cpu_dma_latency

lrwx------. 1 root root 64 Oct 31 15:10 12 -> anon_inode:[eventpoll]

lr-x------. 1 root root 64 Oct 31 15:10 13 -> /dev/urandom

lrwx------. 1 root root 64 Oct 31 15:10 2 -> socket:[16198]

l-wx------. 1 root root 64 Oct 31 15:10 3 -> /var/log/tuned/tuned.log

lrwx------. 1 root root 64 Oct 31 15:10 4 -> socket:[16727]

lrwx------. 1 root root 64 Oct 31 15:10 5 -> anon_inode:[eventfd]

lrwx------. 1 root root 64 Oct 31 15:10 6 -> /tmp/ffi0BhT6y (deleted)

lrwx------. 1 root root 64 Oct 31 15:10 7 -> anon_inode:[eventfd]

lrwx------. 1 root root 64 Oct 31 15:10 8 -> socket:[16749]

lr-x------. 1 root root 64 Oct 31 15:10 9 -> pipe:[16768]

[root@localhost fd]# cat ../cmdline

/usr/bin/python-Es/usr/sbin/tuned-l-P[root@localhost fd]#

If you're wondering how many threads are running:

[root@localhost task]# pwd

/proc/976/task

[root@localhost task]# ll

total 0

dr-xr-xr-x. 6 root root 0 Oct 31 15:34 1178

dr-xr-xr-x. 6 root root 0 Oct 31 15:34 1183

dr-xr-xr-x. 6 root root 0 Oct 31 15:34 1187

dr-xr-xr-x. 6 root root 0 Oct 31 15:34 1191

dr-xr-xr-x. 6 root root 0 Oct 31 15:34 976

ELF

Ececutable and Linkable Format

PID relationships

If child is running but parent has died, it is orphaned

Parents wait for their children
Children die, parents wait for their status
Until the parent receives the status, the child is not truly dead...zombie (defunct)

a zombie process or defunct process is a process that has completed execution but still has an entry in the process table: it is a process in the "Terminated state". Parent processes always wait for their children. If the child process dies, the parent waits for the status. Until this is received, the child is not dead, it is zombie/defunct. The zombied process is not running in user space, it is in the kernel space.

A zombie process can only die if the parent dies.

Double Fork

A process forks and forks again.

This is useful to ensure there's no parent process, preventing zombies. The resulting process becomes a grandchild, not a child.

inodes

stat

can be used to show statistics on a file

[root@localhost task]# ln /etc/resolv.conf /tmp/resolv.conf

[root@localhost task]# ln -s /etc/resolv.conf /tmp/resolv1.conf

[root@localhost task]# stat /etc/resolv.conf

  File: ‘/etc/resolv.conf’

  Size: 68              Blocks: 8          IO Block: 4096   regular file

Device: fd00h/64768d    Inode: 136459736   Links: 2

Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Context: system_u:object_r:net_conf_t:s0

Access: 2018-10-31 15:10:23.341000000 -0400

Modify: 2018-10-31 15:10:23.335000000 -0400

Change: 2018-10-31 16:01:28.755488176 -0400

 Birth: -

[root@localhost task]# stat /tmp/resolv.conf

  File: ‘/tmp/resolv.conf’

  Size: 68              Blocks: 8          IO Block: 4096   regular file

Device: fd00h/64768d    Inode: 136459736   Links: 2

Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Context: system_u:object_r:net_conf_t:s0

Access: 2018-10-31 15:10:23.341000000 -0400

Modify: 2018-10-31 15:10:23.335000000 -0400

Change: 2018-10-31 16:01:28.755488176 -0400

 Birth: -

[root@localhost task]# stat /tmp/resolv2.conf

stat: cannot stat ‘/tmp/resolv2.conf’: No such file or directory

[root@localhost task]# stat /tmp/resolv1.conf

  File: ‘/tmp/resolv1.conf’ -> ‘/etc/resolv.conf’

  Size: 16              Blocks: 0          IO Block: 4096   symbolic link

Device: fd00h/64768d    Inode: 1171458     Links: 1

Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)

Context: unconfined_u:object_r:user_tmp_t:s0

Access: 2018-10-31 16:01:36.422488176 -0400

Modify: 2018-10-31 16:01:36.422488176 -0400

Change: 2018-10-31 16:01:36.424488176 -0400

 Birth: -

limits

Controlled by PAM

When a process is running, /proc/{pid}/limits shows the limits applicable to that PID

troubleshooting - sources of problems

• kernel

• Processes

• Users/Groups

• Permissions

• Space / Inodes

• Capacity (memory / CPU / network IO)

• NSS

• Network

Steps:

1. Verify the problem

2. Read Logs

3. Check permissions

4. Check space

5. Trace

6. Check Ports

7. Trust nobody! (check things again for yourself)

Hostname lookups

use getent - it looks up hostnames the way the system would

(unless you're specifically checking DNS - in which case use dig / nslookup)