Monday, October 04, 2010

Top 10 Java Issues in Production

10 Heap/network issues

Tools: dtrace, hprof, introscope, jconsole, visualvm, yourkit, azul zvision
Invasive tools : bci, jvmti, jvmdi

OS Tools : dtrace, osprofile, vtune

Network/DISK tools: ganglia, iostat, lsof, nagios, netstat, snoop

  • instrumentation is not cheap
  • avoid expensive heap walks
  • thread dumps
  • asynchronous logging
  • finish task and then increment performance counter
  • jconsole is cheap

9 Leaks

symptoms :
  • app consumes all memory
  • heap trend is ramping saw tooth
  • finally Out of Memory
Tools : yourkit, hprof, eclipse mat, jconsole, jhat, jps, visualvm, azul zvision

  • Allotted Vs Live Objects
  • Finalizers, Classloaders
8 I/O

I/O Serialization
Symptoms :
  • Multi-node scale-out does not scale linearly
  • Spending time CPU & IO
Tools :
  • cpu profiling
  • io profiling
  • pick a high performance serialization library
  • avro, krgo
I/O Limits/Tuning

  • Too many file descriptors, cursors etc
  • inconsistent response times
  • Nagios
  • pkg
  • rpm
  • info
  • ulimit
  • yum
  • check OS patches
  • check user and process limits
I/O Sockets, Files, DB

  • socket open/close takes a long time
  • JRMP timeouts, long JDBC calls
  • running out of files, cursors
  • dbms tools, du, iostat, gmon, lsof, netstat
  • ping/telnet tests
7 Locks and Synchronization

  • Adding users/threads/CPUs causes slow down
  • High lock acquire times and contention
  • race conditions, dead locks
  • I/O under load
  • dtrace, lockstat, azulzvision
  • thread dumps
  • IBM visual analyzer ( j.u.c in eclipse )
  • Use non-blocking collections

6 Endless Compilation

  • Time in compiling
  • Time in interpretor
  • -XX:+PrintCompilation
  • CPU profiler

5 Endless Exceptions

  • Application spending time filling stack trace
  • CPU profiler, zvision
  • thread dumps
  • Track caller/callee
  • repeated kill -3
  • don't throw , return
  • JVM's don't optimize exception paths
4 Fragmentation

  • performance degrades over time
  • "Full GC" makes problem go away
  • Lot of free memory, but in tiny fragments
  • GC logging flags, for CMS -XX:PrintFLSStatistics=2 -XX:+PrintCMSInitializationStatistics
  • Fragger
  • Upgrade to latest CMS
  • azul zing and gen pausless GC
  • pooling similar sized and similar aged objects together

3 GC Tuning

  • Entropy(GC) = number of GC flags
  • Too many free parameters
  • 64bit/ large heap no solution
  • Constant 40%-60% CPU utilization by GC
  • Scheduled reboot before full GC
  • Full-time engineer working on GC flags
  • Ask JVM vendor for one flag solution
  • G1 GC, Zing GPGC
2 Spikes in Load

  • Rush hour traffic, tax day, black friday
  • outage under spikes, power law of networks
  • Measure
  • Test with realistic load and realistic multi-node setup
  • build redundancy
1 Versionitis

  • different nodes have different configuration, stack components, versions
  • classpath dist/* , -verbose
  • hard to reproduce
  • Method
  • Version control
  • rigor
0 Collapsing under load

juc profiling

Tuesday, August 26, 2008

Useful commands in Linux and Solaris

  1. truss in solaris , strace in linux

Tuesday, May 06, 2008

Useful Maven commands

checkout a project from subversion

mvn scm:checkout -DconnectionUrl=scm:https://svn.url.project.name -DcheckoutDirectory=name


Java Classloaders

Classloaders are responsible for finding and loading a class.

Class itself is a java object.

Following statements are true,
  1. All classloaders except bootstrap classloaders have a parent classloader
  2. All class loaders are of type java.lang.ClassLoader

Custom class loaders

A custom Class Loader can be created by extending java.lang.ClassLoader.
Two things are important in creating a custom classloader.
  1. Setting parent classloader
  2. implementing findClass() method


Friday, August 24, 2007

Auction site for security vulnerabilities
Wabi Sabi Labi


Monday, July 23, 2007

List of User groups in Bay area.

ACCU  http://www.accu-usa.org/
Bay Area Debian http://bad.debian.net/
BAFUG S.F. http://www.bafug.org/
BALUG http://www.balug.org/
[BAWUG http://www.bawug.org/]
BayPIGgies http://www.baypiggies.net/
BayLISA http://www.baylisa.org/
BayLISA Monitoring SIG
BUUG http://www.buug.org/
CABAL http://linuxmafia.com/cabal/
CalLUG http://www.ocf.berkeley.edu/~linux/
[CCSF LUG mailing list only. Dead site:
Cocoaheads http://cocoaheads.org/us/SiliconValleyCalifornia/
CSUEB LUG http://acc.csueastbay.edu/~linuxug/kwiki/
EBLUG http://www.eblug.org/
Linux Consultants' Lunch inquired about missing mailing list, 2006-02-21.
Linux User Group Hayward (Linuxdojo) http://www.linuxdojo.net/
Linuxstammtisch http://electriclichen.com/linuxbierwanderung/local/#sanfrancisco
NBLUG http://www.nblug.org/
Oakland Perl Mongers http://oakland.pm.org/
PC Clubhouse Linux SIG http://www.pcc.org/sigs/#Linux
PenLUG http://www.penlug.org/
SFBA NetBSD UG http://www.imrryr.org/NetBSD/sfbanug.html
SF LUG http://www.sf-lug.org/
SF OpenBSD UG http://www.sfobug.org/
SF Perl UG http://sf.pm.org/
SF PostgreSQL UG http://pugs.postgresql.org/sfpug/
SVBUG http://www.svbug.com/
SVLUG http://www.svlug.org/
Silicon Valley Perl UG http://www.rf.net/~sv-pm-org/
Silicon Valley Wireless Users and Experimenters http://www.svwux.org/
SJSU LUG http://sjsulug.engr.sjsu.edu/
Smaug http://www.scruz.org/
Sonoma County Sysadmins (SoCoSA) http://socosa.org/
SULUG http://sulug.stanford.edu/
Starship Augusta Ada http://trek.starshine.org/

Linux User Group of Davis (LUGOD) http://www.lugod.org/
Linux User Group of Sacramento (SacLUG) http://www.saclug.org/

Sites of sporadic interest:
Elec. Flea Mkt. 2nd Sat. at De Anza, Mar-Oct: http://www.asvaro.org/fleamarket
Intrusion Detection UG http://idug.cryptojail.net/
Mindsource BOFs http://www.mindsrc.com/
Linux.org events http://www.linux.org/event/
Mozilla Conference
LWCE http://www.linuxworldexpo.com/
Holidays, time changes! see: http://linuxmafia.com/bale/holidays
O'Reilly Open Source Convention, http://conferences.oreillynet.com/os2006/
EFF Bay Chapter http://www.eff.org/bayff/
Bay Area Cypherpunks http://www.cryptorights.org/lists/meetingpunks/
South Bay Community Network sbay.org
Computer History Museum http://www.computerhistory.org/events/
Annual Linux Picnix http://www.linuxpicnic.org/
Stacey's Book Store Events http://www.staceys.com/sanfranciscoevents.html
USENIX Association http://www.usenix.org/events/
CodeCon http://www.codecon.org/
Computers, Freedom, and Privacy http://www.cfp.org/
IEEE Security Symposiums http://www.ieee-security.org/
May 21-24, 2006: http://www.ieee-security.org/TC/SP2006/oakland06.html
Computer Professionals for Social Responsibility http://www.cpsr.org/
Vintage Computer Festival http://www.vintage.org/events.php (November?)
Howard's Events list: http://lg-events.blogspot.com/

Dead groups:
BANG http://www.bang.org/
Internet Developer/Technology Group http://www.inetdevgrp.org/
LBNL LUG http://lug.lbl.gov/
Linux Events http://linevents.com/ (gone)
Linux Movies Group http://www.linuxmovies.org/
SCLUG http://linus.got.net/
sig-beer-west http://www.extasia.org/sig-beer-west/
SlugLUG http://sluglug.ucsc.edu/
WebGuild http://www.webguild.org/

Monday, June 04, 2007

What would happen when tried to compile and run this program?

public class StaticVar
void execute(String b){
static String str;
str = b;

public static void main( String[] args ) {
StaticVar var = new StaticVar();
var.execute( "blah");
1. Compilation Error
2. Runtime Error.
3. Compiles and Runs with no errors.

Answer : 1


Wednesday, May 02, 2007

Nice posting for converting bytes to string and vice versa.


This page is powered by Blogger. Isn't yours?