1) Does it trip when the sum of all 3 phase currents = 600A?
No. It will trip at some time when the load exceeds the breaker rating. If you have 601 amps the breaker would not trip. If you had 675 amps the breaker may trip but it might take hours for it to happen. There are curves that are associated with each breaker and these curves shows how much time it takes for different ampere increments.
2) Does it trip when only one of the 3 phase currents = 600A? It will trip if only on phase of the breaker is overloaded. Same as above.
Check the curves below.
http://ecatalog.squared.com/pubs/Circuit%20Protection/Molded%20Case%20Circuit%20Breakers/Thermal%20Magnetic%20Molded%20Case%20Circuit%20Breakers/FA-FC-FH/0600DB0105.pdf - sorry, link no longer available
Good Answers: